FrequentlyAsked Question / Exact Problem
- “Bills were made yesterday but are not showing in Customer 360.”
- “Customer’s bill was received in CASA but still not showing in C360.”
- “The bill amount was made today but points are not awarded — bill not visible.”
Affected Tenant / Context
Happi Mobiles (APX/ETP integration), Ajfan, Otto, Ibaco, Minister White. Most commonly raised after store-level transactions when the POS → CASA sync pipeline is interrupted.
Root Cause
- ETP/APX sync failure — Happi Mobiles uses an ETP sync process. The urllib3 Python
package was updated to an incompatible version, breaking the /tmp/etp folder sync
process. Bills stacked up and did not flow into CASA. - Queue blockage — Large customer upload jobs (e.g., Rams old store migration) blocked the Kafka processing queue, causing bills for other tenants like Ajfan to queue up and appear delayed.
- Processing delay from Ginesys — For tenants on Ginesys POS, bills may arrive at CASA 3–5 minutes after the actual transaction. If the customer checks immediately after billing, the bill won’t appear.
- CH not synced — In some cases bills are saved in PG (primary DB) but not yet synced to ClickHouse (CH). Customer 360 reads from CH, so the bill appears missing even though it exists.
Common Debugging Steps (For CS Team)
- Note the mobile number, store, and exact bill date/time from the customer.
- Check if this is a Ginesys tenant — if yes, inform the customer that bills take 3–5 minutes to appear after transaction.
- Ask:“Is this affecting all customers at this store, or just one?” — If one, likely a single-record issue. If all, likely a sync failure.
- Check Customer 360 → search mobile → scroll to Bills section. Note if points were also not awarded (confirms bill wasn’t processed).
- If no bill in C360 after 30 minutes, escalate to dev with: mobile number, store code, bill ID, and the date/time of the transaction.
- For Happi Mobiles specifically: check if the ETP folder ( /tmp/etp ) is processing correctly — tag @Renaldo J.
Fix Made & Developer Conclusion
- Happi Mobiles: urllib3 was downgraded to a compatible version. ETP sync resumed.
Bills moved from upload queue to C360 within the hour. - Ajfan (queue blocked): Queue was cleared manually. Bills appeared in C360 after
reprocessing. Suriya flagged that some load spikes can delay processing — the team is
working on priority queue optimization. - Rams data not synced to CH: A prod outage on a specific day caused CH sync to miss 3
numbers. A custom script ( compare-mobile-pg-ch.ts ) was used to identify the gap. Data manually resynced. Alerts were added.
Exact Dev to Reach Out To
Renaldo J — ETP/APX sync, Wondersoft bill coordination
Vishnuj — Bill upload pipeline, Kafka queue
Suriya — Queue issues, CH sync, overall bill flow investigation
Taruun Prasath G S — Customer 360 fixes (paired with Dharshini on C360 migration
work)