Unix timestamp in milliseconds and dtformat

RicRod

Hi,

I'm trying to use GenericCSVData with a Unix timestamp in milliseconds.

I have the following timestamp: 1521590400000, which I got from Binance, using CCXT.
What format should I use for dtformat, since it doesn't fit in the default '%Y-%m-%d %H:%M:%S'?

I'd appreciate if someone could help with this.
Thank you!

backtrader

The format string is given directly to the datetime.datetime.strptime which doesn't support that. The following is additionally implemented. See Docs - Data Feed Reference

      - ``dtformat``: Format used to parse the datetime CSV field. See the
        python strptime/strftime documentation for the format.

        If a numeric value is specified, it will be interpreted as follows

          - ``1``: The value is a Unix timestamp of type ``int`` representing
            the number of seconds since Jan 1st, 1970

          - ``2``: The value is a Unix timestamp of type ``float``

Your timestamp would need to be converted to float dividing by 1000 or be taken as an int discarding the last 3 zeroes.

Your best bet:

Load it with pandas
Use this: https://stackoverflow.com/questions/34883101/pandas-converting-row-with-unix-timestamp-in-milliseconds-to-datetime
Use PandasData as your data feed

RicRod

Thank you very much.

I'll take a look at using pandas.

backtrader

There is something else.You can also pass a callable as the dtformat to convert time. In this case you could pass something like

dtformat=lambda x: datetime.datetime.utcfromtimestamp(float(x) / 1000.0)

From the docs:

        If a **callable** is passed

          - it will accept a string and return a `datetime.datetime` python
            instance

RicRod

@backtrader

Sorry, I'm not entirely sure about the syntax, would dtformat be used like this?

    params = (
        ('nullvalue', float('NaN')),
        dtformat=lambda x: datetime.datetime.utcfromtimestamp(float(x) / 1000.0)
        ('tmformat', '%H:%M:%S'),

        ('datetime', 0),
        ('time', -1),
        ('open', 1),
        ('high', 2),
        ('low', 3),
        ('close', 4),
        ('volume', 5),
        ('openinterest', 6),
    )

Thank you.

backtrader

@ricrod said in Unix timestamp in milliseconds and dtformat:

dtformat=lambda x: datetime.datetime.utcfromtimestamp(float(x) / 1000.0)

Has to look like the others

(dtformat, lambda x: datetime.datetime.utcfromtimestamp(int(x) / 1000.0)),

Changed to int, which probably makes more sense. You could even pass a callable which removes the 3 trailing zeroes and also converts to int.

(dtformat, lambda x: datetime.datetime.utcfromtimestamp(int(x[:-3])),

RicRod

@backtrader

Thank you once again. I will try it.

RicRod

@backtrader
I've tried the following:

class dataFeed(bt.feeds.GenericCSVData):
    params = (
        ('nullvalue', float('NaN')),
        ('dtformat', lambda x: datetime.datetime.utcfromtimestamp(int(x[:-3]))),
        ('tmformat', '%H:%M:%S'),
        
        ('datetime', 0),
        ('time', -1),
        ('open', 1),
        ('high', 2),
        ('low', 3),
        ('close', 4),
        ('volume', 5),
        ('openinterest', -1)
    )

And I get this error:

\Python37-32\lib\site-packages\backtrader\lineseries.py", line 461, in __getattr__
    return getattr(self.lines, name)
AttributeError: 'Lines_LineSeries_DataSeries_OHLC_OHLCDateTime_Abst' object has no attribute '_dtstr'

Dividing by 1000.0 and using float, also gave the same error.
Maybe you know what causes this error?

RicRod

I found a way to solve the error, in this thread:
https://community.backtrader.com/topic/1043/dates-and-time-loaded-from-csv-files-not-precise/2