Data loss in influxdb due to aggregators

[thrinathadevops]

I am pushing data from NATS to InfluxDB using Telegraf. The data is being generated continuously, which results in multiple entries having the same timestamp. My timestamps are in nanoseconds, but in InfluxDB, the data is being aggregated, causing a loss of data points in my Grafana dashboard.

Can any one suggest how to avoid data loss.

example data.

requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test time=1728981942717004500,count=1,errorCount=0,sentBytes=1467,timeStamp=1728981940732,receivedBytes=59275,latency=0,processingTime=0,connectTime=0,responseTime=3151 1728981942717004544
requestsRaw,ObjectType=requestsRaw,errorMessage=noData,errorResponseBody=noData,host=server-3,nodeName=Test-Node,requestName=WebSocket\ request-response\ Sampler,responseCode=200,result=pass,runId=R_20241015-1415,samplerType=transaction,subject=nats.subject.requestsraw,testName=Test latency=0,receivedBytes=59275,time=1728981942717004500,errorCount=0,responseTime=1276,count=1,processingTime=0,sentBytes=1467,timeStamp=1728981942619,connectTime=0 1728981942717004544

Below is my telegraf configuration

[[inputs.nats_consumer]]
servers = [“nats://localhost:4222”]
subjects = [“nats.subject.requestsraw”]
data_format = “xpath_json”
xpath_native_types = true
fieldexclude = [“subjects”, “runId”, “testName”, “nodeName”, “ObjectType”, “errorMessage”, “errorResponseBody”, “requestName”, “responseCode”, “result”, “samplerType”]

[[inputs.nats_consumer.xpath]]
metric_name = “‘requestsRaw’”
metric_selection = “/"
timestamp = “time”
timestamp_format = “unix_ns”
field_selection = "”
[inputs.nats_consumer.xpath.tags]
runId = “string(/runId)”
testName = “string(/testName)”
nodeName = “string(/nodeName)”
ObjectType = “string(/ObjectType)”
errorMessage = “string(/errorMessage)”
errorResponseBody = “string(/errorResponseBody)”
requestName = “string(/requestName)”
responseCode = “string(/responseCode)”
result = “string(/result)”
samplerType = “string(/samplerType)”

[philjb]

I'm not sure what you mean by aggregate (could you explain?) but I believe these docs address what I think is happening to you:

https://docs.influxdata.com/influxdb/v1/write_protocols/line_protocol_tutorial/#duplicate-points

(please close this ticket if so)