[us_dod_disa_3_entities_sourceip_destinationip_network-3f10] Details ============================ Generated On: 2025-01-07 16:36:33 UTC TML Solution DAG Parameters' Details: User Chosen Parametets ---------------------------- STEP 1: Get TML Core Params: `tml_system_step_1_getparams_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - solutionname - us_dod_disa_3_entities_sourceip_destinationip_network-3f10 * - solutiontitle - TML Entity Based Cybersecurity Solution * - solutiondescription - This is an awesome real-time cybersecurity solution built by TSS * - brokerhost - 127.0.0.1 * - brokerport - 9092 * - cloudusername - None * - ingestdatamethod - LOCALFILE STEP 2: Create Kafka Topics: `tml_system_step_2_kafka_createtopic_dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - companyname - Otics * - myname - Sebastian * - myemail - Sebastian.Maurice * - mylocation - Toronto * - replication - 1 * - numpartitions - 1 * - enabletls - 1 * - microserviceid - * - raw_data_topic - iot-raw-data * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_data_topic - ml-data * - prediction_data_topic - prediction-data STEP 3: `Produce to Kafka Topics `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PRODUCETYPE - LOCALFILE * - TOPIC - iot-raw-data * - PORT - _39399 * - IDENTIFIER - TML solution,/rawdata/situng-zeekconn-ldjson.json * - HTTPADDR - https:// * - FROMHOST - ('seb', '127.0.1.1') * - TOHOST - 0.0.0.0 * - CLIENTPORT - Not Applicable * - TSS_CLIENTPORT - Not Applicable * - TML_CLIENTPORT - Not Applicable STEP 4: Preprocesing Data: `tml-system-step-4-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - iot-raw-data * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - preprocessconditions - * - delay - 70 * - array - 0 * - saveasarray - 1 * - topicid - -999 * - rawdataoutput - 1 * - asynctimeout - 120 * - timedelay - 0 * - preprocesstypes - anomprob,trend,avg,min,max * - pathtotmlattrs - --pathtotmlattrs-- * - identifier - TML Entity Based Cybersecurity Detection * - jsoncriteria - uid=_source.destination.address|_source.network.direction|_source.source.address,filter:allrecords~subtopics=_source.destination.bytes,_source.destination.packets,_source.source.bytes,_source.source.packets,_source.network.bytes,_source.network.packets~values=_source.destination.bytes,_source.destination.packets,_source.source.bytes,_source.source.packets,_source.network.bytes,_source.network.packets~identifiers=_source.destination.geo.city_name,_source.destination.port,_source.source.geo.city_name,_source.source.port,_source.network.direction,_source.related.ip~datetime=_source.@timestamp~msgid=_id~latlong=_source.source.geo.location.lat:_source.source.geo.location.lon STEP 4b: Preprocesing Data: `tml-system-step-4b-kafka-preprocess-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - raw_data_topic - --raw_data_topic2-- * - preprocess_data_topic - --preprocess_data_topic2-- * - preprocessconditions - --preprocessconditions2-- * - delay - --delay2-- * - array - --array2-- * - saveasarray - --saveasarray2-- * - topicid - --topicid2-- * - rawdataoutput - --rawdataoutput2-- * - asynctimeout - --asynctimeout2-- * - timedelay - --timedelay2-- * - preprocesstypes - --preprocesstypes2-- * - pathtotmlattrs - --pathtotmlattrs2-- * - identifier - --identifier2-- * - jsoncriteria - --jsoncriteria2-- STEP 5: Entity Based Machine Learning : `tml-system-step-5-kafka-machine-learning-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_data_topic - ml-data * - modelruns - --modelruns-- * - offset - -1 * - islogistic - --islogistic-- * - networktimeout - --networktimeout-- * - modelsearchtuner - --modelsearchtuner-- * - processlogic - --processlogic-- * - dependentvariable - --dependentvariable-- * - independentvariables - --independentvariables-- * - rollbackoffsets - --rollbackoffsets-- * - topicid - -999 * - consumefrom - --consumefrom-- * - fullpathtotrainingdata - --fullpathtotrainingdata-- * - transformtype - --transformtype-- * - sendcoefto - --sendcoefto-- * - coeftoprocess - --coeftoprocess-- * - coefsubtopicnames - --coefsubtopicnames-- STEP 6: Entity Based Predictions: `tml-system-step-6-kafka-predictions-dag `_ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - preprocess_data_topic - iot-preprocess,iot-preprocess2 * - ml_prediction_topic - --ml_prediction_topic-- * - streamstojoin - --streamstojoin-- * - inputdata - --inputdata-- * - consumefrom - --consumefrom2-- * - offset - -1 * - delay - 70 * - usedeploy - --usedeploy-- * - networktimeout - --networktimeout-- * - maxrows - --maxrows-- * - topicid - -999 * - pathtoalgos - --pathtoalgos-- STEP 7: Real-Time Visualization: `tml-system-step-7-kafka-visualization-dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - vipervizport - 49689 * - topic - iot-preprocess,iot-preprocess2 * - dashboardhtml - cybersecuritydetection.html * - secure - 1 * - offset - -1 * - append - 0 * - chip - amd64 * - rollbackoffset - 400 STEP 8: `tml_system_step_8_deploy_solution_to_docker_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Docker Container - maadsdocker/us_dod_disa_3_entities_sourceip_destinationip_network-3f10-amd64 (https://hub.docker.com/r/maadsdocker/us_dod_disa_3_entities_sourceip_destinationip_network-3f10-amd64) * - Docker Run Command - docker run -d -p 5050:5050 -p 4040:4040 -p 6060:6060 \ --env TSS=0 \ --env SOLUTIONNAME=us_dod_disa_3_entities_sourceip_destinationip_network-3f10 \ --env SOLUTIONDAG=solution_preprocessing_dag-us_dod_disa_3_entities_sourceip_destinationip_network-3f10 \ --env GITUSERNAME=smaurice101 \ --env GITREPOURL=https://github.com/smaurice101/raspberrypitss.git \ --env SOLUTIONEXTERNALPORT=5050 \ -v /var/run/docker.sock:/var/run/docker.sock:z \ --env CHIP=amd64 \ --env SOLUTIONAIRFLOWPORT=4040 \ --env SOLUTIONVIPERVIZPORT=6060 \ --env DOCKERUSERNAME='maadsdocker' \ --env EXTERNALPORT=39399 \ --env KAFKACLOUDUSERNAME='MUHRHBPKJYPROKBX' \ --env VIPERVIZPORT=49689 \ --env MQTTUSERNAME='smaurice' \ --env AIRFLOWPORT=9000 \ --env MQTTPASSWORD='' \ --env KAFKACLOUDPASSWORD='' \ --env GITPASSWORD='' \ --env READTHEDOCS='' \ maadsdocker/us_dod_disa_3_entities_sourceip_destinationip_network-3f10-amd64 STEP 9: `tml_system_step_9_privategpt_qdrant_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - PrivateGPT Container - --pgptcontainername-- * - PrivateGPT Run Command - --privategptrun-- * - Qdrant Container - --qdrantcontainer-- * - Qdrant Run Command - --qdrantrun-- * - Consumefrom - --consumefrom-- * - pgpt_data_topic - --pgpt_data_topic-- * - offset - -1 * - rollbackoffset - 400 * - topicid - -999 * - enabletls - 1 * - partition - --partition-- * - prompt - --prompt-- * - context - --context-- * - jsonkeytogather - --jsonkeytogather-- * - keyattribute - --keyattribute-- * - keyprocesstype - --keyprocesstype-- * - vectordbcollectionname - --vectordbcollectionname-- * - concurrency - --concurrency-- * - CUDA_VISIBLE_DEVICES - --cuda-- * - pgpthost - --pgpthost-- * - pgptport - --pgptport-- * - hyperbatch - --hyperbatch-- STEP 10: `tml_system_step_10_documentation_dag `_ ^^^^^^^^^^^^^^^^^^^^^ .. list-table:: * - **User Parameter** - **Chosen Value** * - Solution Documentation URL - https://us_dod_disa_3_entities_sourceip_destinationip_network-3f10.readthedocs.io