How to speed up when add data and preload data?
when add thousands of stocks it will take hours to add and preload data, how can speed it up ? can we save the datas use pickle or other method ,next time when we use , just load it up.
That sounds like a very long time. are you sure you're not running into memory issues? Pandas craps out under memory pressure. It starts compressing and you'll see very high CPU loads but nothing is really getting done.
If parsing the CSV is taking a long time, then I suppose pickling would work, there was a post on that and the guy saw really good improvements, but I think a better solution is probably to maintain a custom data feed in a memory db like Redis.
also check out https://min.io/