Create Real-time OHLC Data with Python WebSocket
23 April 2024
Real-time market data has become crucial for financial applications in the digital era. WebSockets have emerged as a popular technology that enables real-time, full-duplex communication channels between client and server. They are specifically for delivering market data to developers and analysts, making building interactive and dynamic platforms easier.
Why WebSockets?
With WebSockets' rising popularity, it is no wonder that new developers want to experiment with this technology. It is relatively easy to create a server and a client to play around with data using WebSockets, but where the rubber meets the road is when you have access to our Forex API that provides real-time data.
This tutorial will show you how to connect to a WebSocket Server using Python, get real-time data, convert it into OHLC, and log messages received on a File. So, let's dive in!
Prerequisites:
Before getting into detail about this tutorial, you need the following:
1) Python installed
2) Basic knowledge of Python
API Key
Sign up to start your trial and access forex data for 14 days. Your API key can be found in your dashboard. Our comprehensive documentation with example codes is available to guide you through the process.
Lets Code
The code imports data from the TraderMade module to stream real-time forex data. The main objective of the code is to process this streaming data to calculate and store the Open, High, Low, and Close (OHLC) prices for different currency pairs at specified intervals.
The processed data is then printed to the console and saved to a text file.
Here's a breakdown of each part:
Step 1: Installing the TraderMade package
This line installs the TraderMade package with a specific version (0.8.0). The TraderMade package provides access to TraderMade's Forex and Cryptocurrency data through their API.
pip install tradermade==0.8.0
Step 2: Importing necessary modules
1) tradermade.stream: This module provides functionalities to connect to TraderMade's WebSocket for streaming Forex data.
2) JSON: This module is used to parse the JSON data received from the WebSocket.
3) datetime: This module provides functionalities to work with dates and times, which are essential for timestamp manipulation and formatting.
4) defaultdict: This is a subclass of the built-in dictionary. A defaultdict creates a dictionary with a default value type, which is useful for storing OHLC (Open, High, Low, Close) data.
from tradermade import stream import json from datetime import datetime from collections import defaultdict
Step 3: Initializing variables
1) ohlc_data: This nested defaultdict stores OHLC data for different currencies at different time intervals. The nested defaultdict ensures that when a new currency or time interval is encountered, the default values for OHLC are set to 0.
2) previous_interval: This variable stores the previously processed time interval to compare and print the OHLC data for the previous interval.
3) Count: This is a flag that tracks the number of ticks (price updates) received in the current interval.
4) Interval: This is the time interval (in minutes) for calculating the OHLC data. Here, it is set to 2 minutes.
5) Format: This is the format for the timestamp, which is used for rounding and formatting the timestamp.
6) output_file: This is the name of the file to which the OHLC data will be saved.
ohlc_data = defaultdict(lambda: defaultdict(lambda: {'open': 0, 'high': 0, 'low': 0, 'close': 0})) previous_interval = None
count = 0
interval = 2
format = '%Y-%m-%d %H:%M'
output_file = 'ohlc_output.txt'
Step 4: Defining utility functions
round_timestamp()
This function rounds the given timestamp to the nearest minute based on the specified interval. For example, if the interval is set to 2 minutes and the timestamp is 2024-04-22 12:03:45, the rounded timestamp will be 2024-04-22 12:02:00.
It is not advised to use longer timeframes beyond a few minutes directly; instead, create shorter timeframes and aggregate them when querying the files.
def round_timestamp(timestamp, interval_minutes): rounded_minutes = (timestamp.minute // interval_minutes) * interval_minutes rounded_time = timestamp.replace(second=0, microsecond=0, minute=rounded_minutes) return rounded_time
save_to_file()
This function appends the given data to the ohlc_output.txt file. The 'a' mode ensures that the data is appended to the file and not overwritten.
def save_to_file(data): with open(output_file, 'a') as f: f.write(data + '\n')
update_ohlc_data()
This function updates the OHLC data for a given currency at a specific time interval:
1) If the current interval is not in ohlc_data, the OHLC data for that interval and currency is initialized.
2) If the currency is not in the current interval's data, it initializes the OHLC data for that currency.
3) Otherwise, it updates the high, low, and close values based on the received mid-price.
def update_ohlc_data(current_interval, currency, mid):
if current_interval not in ohlc_data:
ohlc_data[current_interval] = defaultdict(lambda: {'open': mid, 'high': mid, 'low': mid, 'close': mid})
elif currency not in ohlc_data[current_interval]:
ohlc_data[current_interval][currency] = {'open': mid, 'high': mid, 'low': mid, 'close': mid}
else:
ohlc_data[current_interval][currency]['high'] = max(ohlc_data[current_interval][currency]['high'], mid)
ohlc_data[current_interval][currency]['low'] = min(ohlc_data[current_interval][currency]['low'], mid)
ohlc_data[current_interval][currency]['close'] = mid
process_interval()
This function processes the OHLC data for the previous interval, prints it, and saves it to a file:
1) It checks if there is a previous interval, if the count is 1 (indicating that at least one tick was received in the current interval) and if the previous interval differs from the current interval.
2) If these conditions are met, it iterates over the currencies and their respective OHLC data for the previous interval, formats the data, prints it to the console, and saves it to the output file.
3) After processing, it resets the count to 0 and deletes the previous_interval.
def process_interval(current_interval):
global previous_interval, count
if len(ohlc_data) > 1:
for currency, ohlc in ohlc_data[previous_interval].items():
output = f"{previous_interval:<10} {currency:<10} {ohlc['open']:<10.5f} {ohlc['high']:<10.5f} {ohlc['low']:<10.5f} {ohlc['close']:<10.5f}"
print(output)
save_to_file(output)
count = 0
del ohlc_data[previous_interval]
process_data()
This function processes the data received from the WebSocket:
1) It checks if the received data is not "Connected" (indicating that it is actual data and not a connection message).
2) It parses the JSON data to extract the timestamp, currency symbol, bid, and ask prices.
3) It calculates the mid-price as the average of the bid and ask prices.
4) It updates the OHLC data using the update_ohlc_data() function.
5) It processes the interval using the process_interval() function.
6) If the count is 0 (indicating the start of a new interval), it updates the previous interval and sets the count to 1.
7) If there is any error during the data processing, it prints an error message.
def process_data(data):
global previous_interval, count
if data != "Connected":
try:
data = json.loads(data)
timestamp = datetime.fromtimestamp(int(data['ts']) / 1000)
current_interval = round_timestamp(timestamp, interval).strftime(format)
currency = data['symbol']
mid = round(float((data['bid']) + float(data['ask'])) / 2, 5)
update_ohlc_data(current_interval, currency, mid)
process_interval(current_interval)
if count == 0:
previous_interval = current_interval
count = 1
except Exception as e:
print(f"Error processing data: {e}")
else:
print("Connected to WebSocket")
Step 5: Setting up and connecting to the WebSocket
1) Stream.set_ws_key(): Sets the WebSocket key to connect to TraderMade's WebSocket. It would help if you replaced the placeholder with your actual API key.
2) Stream.set_symbols(): This sets the symbols for the data that should be streamed. Here, it is set to "USDJPY, EURGBP."
3) stream.stream_data(): Starts streaming data and calls the process_data() function as a callback.
4) Stream. Connect (): Connects to the WebSocket and starts streaming data.
stream.set_ws_key("API_KEY") # Replace with your actual API key stream.set_symbols("USDJPY,EURGBP")
stream.stream_data(process_data)
stream.connect()
Execute the program
The command "python websocket_ohlc.py" runs a Python script named websocket_ohlc.py. The script's name can be changed, but the extension.py indicates it is a Python program. This script uses WebSockets to calculate and save real-time OHLC (Open, High, Low, Close) market data into a file, making it suitable for financial applications.
python websocket_ohlc.py
Hurray! Our output is saved in the file “ ohlc_output.txt”. So, there you have it—live data is now streaming in, and OHLC is calculated. The code will create a text file in the folder where you saved it.
Full code
from tradermade import stream
import json
from datetime import datetime
from collections import defaultdict
ohlc_data = defaultdict(lambda: defaultdict(lambda: {'open': 0, 'high': 0, 'low': 0, 'close': 0}))
previous_interval = None
count = 0
interval = 1
format = '%Y-%m-%d %H:%M'
output_file = 'ohlc_output.txt'
def round_timestamp(timestamp, interval_minutes):
rounded_minutes = (timestamp.minute // interval_minutes) * interval_minutes
rounded_time = timestamp.replace(second=0, microsecond=0, minute=rounded_minutes)
return rounded_time
def save_to_file(data):
with open(output_file, 'a') as f:
f.write(data + '\n')
def update_ohlc_data(current_interval, currency, mid):
if current_interval not in ohlc_data:
ohlc_data[current_interval] = defaultdict(lambda: {'open': mid, 'high': mid, 'low': mid, 'close': mid})
elif currency not in ohlc_data[current_interval]:
ohlc_data[current_interval][currency] = {'open': mid, 'high': mid, 'low': mid, 'close': mid}
else:
ohlc_data[current_interval][currency]['high'] = max(ohlc_data[current_interval][currency]['high'], mid)
ohlc_data[current_interval][currency]['low'] = min(ohlc_data[current_interval][currency]['low'], mid)
ohlc_data[current_interval][currency]['close'] = mid
def process_interval(current_interval):
global previous_interval, count
if len(ohlc_data) > 1:
for currency, ohlc in ohlc_data[previous_interval].items():
output = f"{previous_interval:<10} {currency:<10} {ohlc['open']:<10.5f} {ohlc['high']:<10.5f} {ohlc['low']:<10.5f} {ohlc['close']:<10.5f}"
print(output)
save_to_file(output)
count = 0
del ohlc_data[previous_interval]
def process_data(data):
global previous_interval, count
if data != "Connected":
try:
data = json.loads(data)
timestamp = datetime.fromtimestamp(int(data['ts']) / 1000)
current_interval = round_timestamp(timestamp, interval).strftime(format)
currency = data['symbol']
mid = round(float((data['bid']) + float(data['ask'])) / 2, 5)
update_ohlc_data(current_interval, currency, mid)
process_interval(current_interval)
if count == 0:
previous_interval = current_interval
count = 1
except Exception as e:
print(f"Error processing data: {e}")
else:
print("Connected to WebSocket")
stream.set_ws_key("API_KEY") # Replace with your actual API key
stream.set_symbols("USDJPY,EURGBP")
stream.stream_data(process_data)
stream.connect()
Final Thoughts
Feel free to amend this function if you want a different output by parsing JSON and setting a custom timestamp format. Also, some work may be needed to deal with reconnection and handling missed data while the service is down. One solution is to save data in a file and delete it after a few hours or a day. We hope this tutorial has been helpful.
Please get in touch with our experts for any technical questions through live chat or the support@tradermade.com email address. We are always looking forward to your feedback.
