Sophos Central Defined Web Categories not syncing correctly to XG Firewalls

In Central is a Taskqueue. Do you see the sync there? 

I get a success but looking at the sync log, something doesn't look right

{
  "opcodeID": 1,
  "entityID": 502,
  "entityName": "create_category",
  "opcodeType": 1,
  "orderID": 0,
  "opcodeString": "",
  "responseStatus": "{\"Event\":\"ADD\",\"statusmessage\":\"some detail failed\",\"Entity\":\"webfiltercategory\",\"status\":\"201\"}",
  "uniqueName": "Footwear Manufacturers-502",
  "updateFlag": "f",
  "mainEntity": "t"
}

{
  "opcodeID": 2,
  "entityID": 502,
  "entityName": "manage_category",
  "opcodeType": 1,
  "orderID": 1,
  "opcodeString": "",
  "responseStatus": "{\"status\":\"201\",\"statusmessage\":\"some detail failed\",\"Entity\":\"webfiltercategory\",\"Event\":\"UPDATE\"}",
  "uniqueName": "Footwear Manufacturers-502",
  "updateFlag": "t",
  "mainEntity": "t"
}

A snippet from csc.log on one of the boxes.

MESSAGE   Aug 18 09:10:32  [worker:7061]: {"request":{"method":"opcode","name":"apiInterface","version":"1.0","type":"json","length":609,"data":{ "webcattype": "110", "urltype": "1", "timestamp": "1586169190000
", "name": "Footwear Manufacturers", "cmbAttribute": "3", "APIVersion": "1502.0", "urls": [ "newbalance.com", "adidas.com", "nike.com", "asics.com" ], "keywords": [ "" ], "desc": "", "deniedmsg": "", "httpactio
ns": "", "action": "", "httpsactions": "", "mode": "313", "cmbBWPolicy": "None", "webcategoryid": "Footwear Manufacturers", "contenttype": "1", "chkdeniedmsg": "on", "Entity": "webfiltercategory", "isDefault":
"0", "Event": "ADD", "___component": "Central Management", "currentlyloggedinuserip": "127.0.0.1", "___username": "admin" }}}
MESSAGE   Aug 18 09:10:33  [worker:7060]: {"request":{"method":"opcode","name":"create_category","version":"1.6","type":"json","length":636,"data":{ "httpactions": "", "name": "Footwear Manufacturers", "keyword
s": [ ], "mode": "313", "isdefault": "0", "contenttype": "1", "___username": "admin", "cmbAttribute": "3", "type": "0", "currentlyloggedinuserip": "127.0.0.1", "APIVersion": "1502.0", "timestamp": "158616919000
0", "___component": "Central Management", "chkdeniedmsg": "on", "currentlyloggedinuserid": 3, "isuploadrestriction": "0", "Entity": "webfiltercategory", "httpsactions": "", "desc": "", "acltype": "0", "urltype"
: "1", "webcattype": "110", "cmbBWPolicy": "None", "deniedmsg": "", "urls": [ "newbalance.com", "adidas.com", "nike.com", "asics.com" ], "Event": "ADD" }}}
MESSAGE   Aug 18 09:10:33  [worker:6763]: {"request":{"method":"opcode","name":"apiInterface","version":"1.6","type":"json","length":215,"data":{ "operation": "create", "mode": "333", "urltype": "1", "urls": [
"newbalance.com", "adidas.com", "nike.com", "asics.com" ], "___caller": "system", "contenttype": "1", "webcategoryid": 1043, "APIVersion": "1500.1" }}}
MESSAGE   Aug 18 09:10:33  [worker:3737]: {"request":{"method":"opcode","name":"add_url_to_webcat","version":"1.6","type":"json","length":215,"data":{ "contenttype": "1", "APIVersion": "1500.1", "urls": [ "newb
alance.com", "adidas.com", "nike.com", "asics.com" ], "mode": "333", "operation": "create", "___caller": "system", "webcategoryid": 1043, "urltype": "1" }}}


 MODE:333 FILE NOT FOUND
WARNING   Aug 18 09:10:34  [create_category:7060]: Action with NOFAIL Failed.

 PAckage ::::webfilter::webfiltercategoryMESSAGE   Aug 18 09:10:34  [worker:7157]: {"request":{"method":"opcode","name":"apiInterface","version":"1.0","type":"json","length":612,"data":{ "APIVersion": "1502.0",
 "webcattype": "110", "urltype": "1", "timestamp": "1586169190000", "name": "Footwear Manufacturers", "cmbAttribute": "3", "urls": [ "newbalance.com", "adidas.com", "nike.com", "asics.com" ], "keywords": [ "" ]
, "desc": "", "deniedmsg": "", "httpactions": "", "action": "", "httpsactions": "", "mode": "314", "cmbBWPolicy": "None", "webcategoryid": "Footwear Manufacturers", "contenttype": "1", "chkdeniedmsg": "on", "Ev
ent": "UPDATE", "Entity": "webfiltercategory", "isDefault": "0", "___component": "Central Management", "currentlyloggedinuserip": "127.0.0.1", "___username": "admin" }}}
MESSAGE   Aug 18 09:10:35  [worker:7151]: {"request":{"method":"opcode","name":"manage_category","version":"1.6","type":"json","length":700,"data":{ "contenttype": "1", "httpactions": "", "acltype": "0", "type"
: "0", "urltype": "1", "APIVersion": "1502.0", "isuploadrestriction": "0", "currentlyloggedinuserip": "127.0.0.1", "deniedmsg": "", "isDefault": "0", "desc": "", "Entity": "webfiltercategory", "urls": [ "newbal
ance.com", "adidas.com", "nike.com", "asics.com" ], "isdefault": "0", "currentlyloggedinuserid": 3, "___username": "admin", "httpsactions": "", "Event": "UPDATE", "keywords": [ ], "webcattype": "110", "mode": "
314", "cmbAttribute": "3", "chkdeniedmsg": "on", "cmbBWPolicy": "None", "timestamp": "1586169190000", "___component": "Central Management", "name": "Footwear Manufacturers", "webcategoryid": "Footwear Manufactu
rers" }}}
ERROR     Aug 18 09:10:35  [manage_category:7151]: csc_av_fetch: 'input[1]' invalid type supplied(5) required(1)
MESSAGE   Aug 18 09:10:35  [ips:1070]: {"ips":{"method":"nservice","name":"ips:webconfig","version":"1.6","type":"text","length":0}}
MESSAGE   Aug 18 09:10:35  [awarrenhttp:1062]: {"awarrenhttp":{"method":"nservice","name":"awarrenhttp:reload","version":"1.6","type":"text","length":0}}
MESSAGE   Aug 18 09:10:35  [worker:7144]: {"request":{"method":"opcode","name":"apiInterface","version":"1.6","type":"json","length":181,"data":{ "webcategoryid": 1043, "operation": "update", "___caller": "syst
em", "contenttype": "1", "urls": [ "adidas.com", "nike.com", "asics.com" ], "mode": "333", "APIVersion": "1500.1" }}}
MESSAGE   Aug 18 09:10:35  [worker:7143]: {"request":{"method":"opcode","name":"add_url_to_webcat","version":"1.6","type":"json","length":181,"data":{ "APIVersion": "1500.1", "operation": "update", "mode": "333
", "___caller": "system", "webcategoryid": 1043, "urls": [ "adidas.com", "nike.com", "asics.com" ], "contenttype": "1" }}}

Detail I forgot to add . Everything else seems to sync correctly .

Ip Host entries , FQDN , FQDN groups etc etc

Could you inspect the applog.log, if any error at this time is found? 

No errors matching for that time

Closest I got is this.

Really a bit puzzled by this tbqh.Have logged a case with Sophos for remote tshooting.

Aug 11 13:21:28 fwcm-updaterd:status RUNNING Status: 500 Error: Opcode Failed
Aug 11 13:31:49 update_tls_error_whitelist
Aug 11 13:33:00 update_tls_error_whitelist
Aug 11 13:33:07 update_tls_error_whitelist
Aug 12 12:00:42 update_tls_error_whitelist
Aug 16 11:35:29 Event Array is ::HASH(0x9fb9690) HASH(0x9fb9680)Aug 16 11:35:30 update_tls_error_whitelist
Aug 16 11:41:43 update_tls_error_whitelist
Aug 16 12:16:34 update_tls_error_whitelist
Aug 16 13:38:40 update_tls_error_whitelist
Aug 16 16:56:35 img2datasrc: /conf/httpclient/customizeimages/default/ico_status_error.png (image/png)
Aug 16 16:56:35 errorpage.html rebuild
Aug 16 16:56:35 img2datasrc: /conf/httpclient/customizeimages/default/ico_status_error.png (image/png)
Aug 16 16:56:35 errorpage.html rebuilt
Aug 16 16:56:36 img2datasrc: /conf/httpclient/customizeimages/default/ico_status_error.png (image/png)
Aug 16 16:56:36 connerrorpage.html rebuild
Aug 16 16:56:36 img2datasrc: /conf/httpclient/customizeimages/default/ico_status_error.png (image/png)
Aug 16 16:56:36 connerrorpage.html rebuilt
Aug 16 16:57:02 fwcm-heartbeatd:status UNTOUCHED Status: 500 Error: Opcode Failed
Aug 16 16:57:03 fwcm-updaterd:status UNTOUCHED Status: 500 Error: Opcode Failed
Aug 16 16:57:03 fwcm-eventd:status UNTOUCHED Status: 500 Error: Opcode Failed

Check the applog.log for this particular time frame. 

Hi Toni I think we've figured it out with a bit of thanks to you.

We have some External Threatfeed URL's that we parse into web cats and it looks like one of them seems to be the culprit.

Taking off one of the feeds results in entries starting to populate again properly.Why it allows the creation of the web category and creation of a single entry is a bit beyond me.

The culprit list is over 30 000 lines long.We initially thought there must be a logic/domain mistake deleting and re-importing the url results in no bad domains in /log/nSXLd/ *

Kinda feels like there is a limit of max URL's that can be imported or parsed ? Seems to be consistent between beefy virtual boxes and hardware appliances.

We'd get messages like the following in /log/nSXLd.log

[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zrw1.destinia.jp
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zsbamj.a1.ro
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zsi7.destinia.do
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: ztqnls.lojasrenner.com.br

Hi Toni I think we've figured it out with a bit of thanks to you.

We have some External Threatfeed URL's that we parse into web cats and it looks like one of them seems to be the culprit.

Taking off one of the feeds results in entries starting to populate again properly.Why it allows the creation of the web category and creation of a single entry is a bit beyond me.

The culprit list is over 30 000 lines long.We initially thought there must be a logic/domain mistake deleting and re-importing the url results in no bad domains in /log/nSXLd/ *

Kinda feels like there is a limit of max URL's that can be imported or parsed ? Seems to be consistent between beefy virtual boxes and hardware appliances.

We'd get messages like the following in /log/nSXLd.log

[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zrw1.destinia.jp
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zsbamj.a1.ro
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: zsi7.destinia.do
[2021-08-19 12:01:36] <140096406185152> [error] nSXLd: Custom category URL not found: ztqnls.lojasrenner.com.br