date: 2016-03-09
First you enable ticket policy.
For some reason somebody forgot to include tickets in version 2 of the API, you can only see ticket data with version 1 of the API, and you cannot set the truncate limit. That means you need to program extra logic to iterate over the entire dataset burning up API requests. Every smart person I spoke to regarding this cannot understand this illogical step in the improvement of their product. To top it off, try and download the KB data, and the API may say no… but the web portal says yes.
Well I was left with no choice but to do the horrible act of webscraping, thankfully I was able to use the python extension in visual studio and take advantage of some intellisence. So after installing selenium python libraries and firefox, I was able to make some progress and build up a job for grabbing the ticket and KB data in one request’ish.
I’m not a data analyst in fact it’s not in my personality, I can do it, but it’s just not fun. So thanks to Andrew for the recommendation to use Talend Studio to do the legwork, I mean every time I use it, I realise just how awesome it is.
So then the plan is to use my python script to webscrape some data, then in talend studio I call that script, make a webrequest to the API for some other data, and then with some data mapping and a custom IP to host mapping file I can pull all this data together into a nice big flatfile for Management. Of course I’m a few KPI short but not bad for a start.
My folder structure is;
D:\Data\Qualys
\Xml
\maps
\ExternalJob
\Documentation
The Python code is below;
# Requirements
#
# Install Python 2.7.9 or greater, added to path variable.
# Install additional python libraries;
# C:\Python27\Scripts\pip.exe install selenium
# C:\Python27\Scripts\pip.exe install bcrypt
# Install firefox 43.0.1 or higher
# Invoke this script from talend studio job; C:\Python27\python.exe D:\Data\Qualys\ExternalJob\QualysDownloadSelenium.py
# Created 22-02-2016
# Last Tested 09-03-2016
# Script likely to break upon password expiry and or changes to the qualys site
# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import unittest, time, re, os
import base64
SAVE_TO_DIRECTORY = “D:\\Data\\Qualys\\Xml\\” # change this to the correct path
userAccount = “Z29kbW9kZQ==” # change for real account, base64 to stop shoulder snoops
Password = “dTgwMDhwYXNzd2Rub3RsaWtlbHl0b2JlaGVyZS4uLi4uMTIz” # change for real password, base64 to stop shoulder snoops
tmpAcc = base64.b64decode(userAccount)
spltUser = tmpAcc.split(r’-‘)
tmpAcc = “”
tmpKBfile = SAVE_TO_DIRECTORY+”DL_vulnerabilities_“+spltUser[0]+”_“+spltUser[1]+”_“+time.strftime(“%Y%m%d”)+”.xml”
tmpTLfile = SAVE_TO_DIRECTORY+”DL_tickets_“+spltUser[0]+”_“+spltUser[1]+”_“+time.strftime(“%Y%m%d”)+”.xml”
docName = “2tmpVulns”
docName2 = “2tmpTickets”
# check for files we don’t want to trip over
if os.path.isfile(SAVE_TO_DIRECTORY+docName+”.xml”):
os.remove(SAVE_TO_DIRECTORY+docName+”.xml”)
if os.path.isfile(SAVE_TO_DIRECTORY+docName2+”.xml”):
os.remove(SAVE_TO_DIRECTORY+docName2+”.xml”)
if os.path.isfile(tmpKBfile):
os.remove(tmpKBfile)
if os.path.isfile(tmpTLfile):
os.remove(tmpTLfile)
profile = webdriver.FirefoxProfile()
profile.set_preference(“browser.download.folderList”, 2)
profile.set_preference(“browser.download.dir”, SAVE_TO_DIRECTORY)
profile.set_preference(“browser.download.manager.alertOnEXEOpen”, False);
profile.set_preference(“browser.helperApps.neverAsk.saveToDisk”, “text/xml, application/msword, application/csv, application/ris, text/csv, image/png, application/pdf, text/html, text/plain, application/zip, application/x-zip, application/x-zip-compressed, application/download, application/octet-stream” )
profile.set_preference(“browser.helperApps.neverAsk.openFile”, “text/xml, application/msword, application/csv, application/ris, text/csv, image/png, application/pdf, text/html, text/plain, application/zip, application/x-zip, application/x-zip-compressed, application/download, application/octet-stream” )
profile.set_preference(“browser.download.manager.showWhenStarting”, False);
profile.set_preference(“browser.download.manager.focusWhenStarting”, False);
profile.set_preference(“browser.download.useDownloadDir”, True);
profile.set_preference(“browser.helperApps.alwaysAsk.force”, False);
profile.set_preference(“browser.download.manager.alertOnEXEOpen”, False);
profile.set_preference(“browser.download.manager.closeWhenDone”, True);
profile.set_preference(“browser.download.manager.showAlertOnComplete”, False);
profile.set_preference(“browser.download.manager.useWindow”, False);
profile.set_preference(“services.sync.prefs.sync.browser.download.manager.showWhenStarting”, False);
profile.set_preference(“pdfjs.disabled”, True);
class QualysSeleniumScript(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox(firefox_profile=profile)
self.driver.implicitly_wait(30)
self.base_url = “https://qualysguard.qualys.eu”
self.verificationErrors = []
self.accept_next_alert = True
def test_qualys_selenium_script(self):
driver = self.driver
driver.get(self.base_url + “/”)
driver.find_element_by_id(“myform_UserLogin”).clear()
driver.find_element_by_id(“myform_UserLogin”).send_keys(base64.b64decode(userAccount))
driver.find_element_by_id(“myform_UserPasswd”).clear()
driver.find_element_by_id(“myform_UserPasswd”).send_keys(base64.b64decode(Password))
driver.find_element_by_name(“_form_action1”).click()
driver.get(self.base_url + “/fo/tools/kbase.php”)
time.sleep(3)
driver.find_element_by_id(“ext-gen202”).click()
driver.find_element_by_id(“ext-gen248”).click()
for i in range(60):
try:
if self.is_element_present(By.ID, “format_handle”): break
except: pass
time.sleep(1)
else: self.fail(“time out”)
driver.find_element_by_id(“select_download_format_xml”).click()
driver.find_element_by_id(“dload_btn”).click()
# should loop around watching for finsihed download KB
# DL_vulnerabilities_mdand_mk_20160309.xml
# DL_vulnerabilities_mdand_mk_20160309.xml.part
X = 0
while True:
time.sleep(5)
X += 1
if os.path.isfile(tmpKBfile):
if os.stat(tmpKBfile).st_size > 0:
break
if X > 60:
break # 60 iterations of 5 seconds will be 5 minutes
# rename downloaded file for KB.
os.chdir(SAVE_TO_DIRECTORY)
if os.path.isfile(docName+”.xml”):
os.remove(docName+”.xml”)
files = filter(os.path.isfile, os.listdir(SAVE_TO_DIRECTORY))
files = [os.path.join(SAVE_TO_DIRECTORY, f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
newest_file = files[-1]
os.rename(newest_file, docName+”.xml”)
#Now we pull down the tickets because the API is poo poo
driver.get(self.base_url + “/fo/remedy/index.php”)
time.sleep(3)
driver.find_element_by_id(“ext-gen68”).click() #search button
time.sleep(3)
driver.find_element_by_id(“dl[search][state_search][OPEN]”).click()
driver.find_element_by_id(“dl[search][state_search][RESOLVED]”).click()
driver.find_element_by_id(“dl[search][state_search][CLOSED]”).click()
driver.find_element_by_id(“dl[search][state_search][IGNORED]”).click()
driver.find_element_by_id(“search_btn”).click()
time.sleep(6)
driver.find_element_by_id(“ext-gen65”).click() #new button
driver.find_element_by_id(“ext-gen114”).click() #new menu download…
driver.find_element_by_id(“select_download_format_xml”).click()
driver.find_element_by_id(“dload_btn”).click()
# need to do loop looking for completed download
# DL_tickets_mdand_mk_20160309.xml
# DL_tickets_mdand_mk_20160309.xml.part
X = 0
while True:
time.sleep(5)
X += 1
if os.path.isfile(tmpTLfile):
if os.stat(tmpTLfile).st_size > 0:
break
if X > 60:
break # 60 iterations of 5 seconds will be 5 minutes #check file is greater than 0kb
#»to do rename downloaded file.
os.chdir(SAVE_TO_DIRECTORY)
if os.path.isfile(docName2+”.xml”):
os.remove(docName2+”.xml”)
files = filter(os.path.isfile, os.listdir(SAVE_TO_DIRECTORY))
files = [os.path.join(SAVE_TO_DIRECTORY, f) for f in files] # add path to each file
files.sort(key=lambda x: os.path.getmtime(x))
newest_file = files[-1]
os.rename(newest_file, docName2+”.xml”)
#finish off by logging off
driver.find_element_by_id(“ext-gen78”).click()
time.sleep(5)
def is_element_present(self, how, what):
try: self.driver.find_element(by=how, value=what)
except NoSuchElementException as e: return False
return True
def is_alert_present(self):
try: self.driver.switch_to_alert()
except NoAlertPresentException as e: return False
return True
def close_alert_and_get_its_text(self):
try:
alert = self.driver.switch_to_alert()
alert_text = alert.text
if self.accept_next_alert:
alert.accept()
else:
alert.dismiss()
return alert_text
finally: self.accept_next_alert = True
def tearDown(self):
self.driver.quit()
self.assertEqual([], self.verificationErrors)
if __name__ == “__main__”:
unittest.main()
Now it’s time to document the Talend Studio job, which just a few click…. see below;
Job Documentation
Generated by Talend Open Studio for Big Data
Project Name
Qualys
GENERATION DATE
09-Mar-2016 16:06:01
AUTHOR
user@talend.com
Talend Open Studio VERSION
6.1.1.20151214_1327
Summary
Project Description
Description
Preview Picture
Settings
Context List
Component List
Components Description
Project Description
Properties
Values
Name
Qualys
Language
java
Description
Description
Properties
Values
Name
QualysDataProcessor
Author
user@talend.com
Version
0.1
Purpose
XML TO CSV
Status
Description
Creation
14-Jan-2016 14:52:57
Modification
09-Mar-2016 16:06:00
Preview Picture
Settings
Extra settings
Name
Value
COMP_DEFAULT_FILE_DIR
Multi thread execution
false
Implicit tContextLoad
false
Status & Logs
Name
Value
Use statistics (tStatCatcher)
false
Use logs (tLogCatcher)
false
Use volumetrics (tFlowMeterCatcher)
false
On Console
false
On Files
false
On Databases
false
Catch components statistics
false
Catch runtime errors
true
Catch user errors
true
Catch user warnings
true
Context List
ContextDefault
Name
Prompt
Need Prompt?
Type
Value
Source
Password
Password?
false
id_Password
****
ContextVars
Component List
Component Name
Component Type
tFileInputDelimited_1
tFileInputDelimited
tFileInputXML_1
tFileInputXML
tFileInputXML_2
tFileInputXML
tFileInputXML_5
tFileInputXML
tFileOutputDelimited_2
tFileOutputDelimited
tHttpRequest_2
tHttpRequest
tSystem_1
tSystem
tSystem_2
tSystem
tXMLMap_1
tXMLMap
Components Description
Component tFileInputDelimited
UNIQUE NAME
tFileInputDelimited_1
INPUT(S)
none
LABEL
IP2DeptMap
OUTPUT(S)
tXMLMap_1
Component Parameters:
Properties
Values
Unique Name
tFileInputDelimited_1
Component Name
tFileInputDelimited
Version
0.102 (ALPHA)
Family
File/Input
Start
false
Startable
true
SUBTREE_START
false
END_OF_FLOW
false
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tFileInputDelimited
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
!!!FILENAMETEXT.NAME!!!
“When the input source is a stream or a zip file,footer and random shouldn’t be bigger than 0.”
File name/Stream
“D:/Data/Qualys/maps/Host-to-Department.csv”
CSV options
false
Row Separator
”\n”
CSV Row Separator
”\n”
Field Separator
”,”
Escape char
”””
Text enclosure
”””
Header
1
Footer
0
Limit
Skip empty rows
false
Uncompress as zip file
false
Die on error
false
REPOSITORY_ALLOW_AUTO_SWITCH
false
Schema
!!!SCHEMA_REJECT.NAME!!!
null:errorCode errorMessage
!!!TEMP_DIR.NAME!!!
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace”
Advanced separator (for numbers)
false
Thousands separator
”,”
Decimal separator
”.”
Extract lines at random
false
Number of lines
10
Trim all columns
false
Check column to trim
[{TRIM=false, SCHEMA_COLUMN=IP}, {TRIM=false, SCHEMA_COLUMN=CIDR}, {TRIM=false, SCHEMA_COLUMN=Zone}, {TRIM=false, SCHEMA_COLUMN=Department}, {TRIM=false, SCHEMA_COLUMN=Function}]
Check each row structure against schema
false
Check date
false
Encoding
“UTF-8”
Split row before field
false
Permit hexadecimal (0xNNN) or octal (0NNNN) for numeric types
false
Decode table
[{DECODE=false, SCHEMA_COLUMN=IP}, {DECODE=false, SCHEMA_COLUMN=CIDR}, {DECODE=false, SCHEMA_COLUMN=Zone}, {DECODE=false, SCHEMA_COLUMN=Department}, {DECODE=false, SCHEMA_COLUMN=Function}]
!!!DESTINATION.NAME!!!
Min column number of optimize code
100
Label format
IP2DeptMap
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for metadata :
Column
Key
Type
Length
Precision
Nullable
Comment
IP
false
String
11
true
CIDR
false
String
14
true
Zone
false
String
8
true
Department
false
String
3
true
Function
false
String
15
true
Original Function Parameters:
Component tFileInputXML
UNIQUE NAME
tFileInputXML_1
INPUT(S)
tHttpRequest_2
LABEL
WsTicketList
OUTPUT(S)
tXMLMap_1
Component Parameters:
Properties
Values
Unique Name
tFileInputXML_1
Component Name
tFileInputXML
Version
0.102 (ALPHA)
Family
| File/Input | XML |
Start
false
Startable
true
SUBTREE_START
true
END_OF_FLOW
false
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tFileInputXML
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH
false
!!!SCHEMA.NAME!!!
File name/Stream
“D:/Data/Qualys/Xml/2tmpTickets.xml”
Loop XPath query
“/DATALIST/LIST/RECORD”
Mapping
[{QUERY=”KEY[@name=\“TICKET_#\”]”, NODECHECK=, SCHEMA_COLUMN=NUMBER}, {QUERY=”KEY[@name=\“STATE\”]”, NODECHECK=, SCHEMA_COLUMN=STATE}, {QUERY=”KEY[@name=\“DUE_DATE\”]”, NODECHECK=, SCHEMA_COLUMN=DUE_DATE}, {QUERY=”KEY[@name=\“IP\”]”, NODECHECK=, SCHEMA_COLUMN=IP}, {QUERY=”KEY[@name=\“PORT_#\”]”, NODECHECK=, SCHEMA_COLUMN=PORT}, {QUERY=”KEY[@name=\“DNS_HOSTNAME\”]”, NODECHECK=, SCHEMA_COLUMN=DNS_HOSTNAME}, {QUERY=”KEY[@name=\“SEVERITY\”]”, NODECHECK=, SCHEMA_COLUMN=SEVERITY}, {QUERY=”KEY[@name=\“QID\”]”, NODECHECK=, SCHEMA_COLUMN=QID}, {QUERY=”KEY[@name=\“VULNERABILITY_TITLE\”]”, NODECHECK=, SCHEMA_COLUMN=VULNERABILITY_TITLE}, {QUERY=”KEY[@name=\“MODIFIED\”]”, NODECHECK=, SCHEMA_COLUMN=MODIFIED}, {QUERY=”KEY[@name=\“CREATED\”]”, NODECHECK=, SCHEMA_COLUMN=CREATED}, {QUERY=”KEY[@name=\“RESOLVED\”]”, NODECHECK=, SCHEMA_COLUMN=RESOLVED}]
Limit
-1
Die on error
false
!!!SCHEMA_REJECT.NAME!!!
null:errorCode errorMessage
Advanced separator (for numbers)
false
Thousands separator
”,”
Decimal separator
”.”
Ignore the namespaces
false
Ignore DTD file
false
Generate a temporary file
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml”
Use Separator for mode Xerces
false
Field Separator
”,”
Generation mode
Dom4j
Validate date
false
Encoding
“UTF-8”
Min column number of optimize code
100
Label format
WsTicketList
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for metadata :
Column
Key
Type
Length
Precision
Nullable
Comment
NUMBER
false
Integer
6
true
STATE
false
String
12
true
DUE_DATE
false
String
20
true
IP
false
String
15
true
PORT
false
String
5
true
DNS_HOSTNAME
false
String
37
true
SEVERITY
false
String
46
true
QID
false
Integer
6
true
VULNERABILITY_TITLE
false
String
197
true
MODIFIED
false
String
28
true
CREATED
false
String
28
true
RESOLVED
false
String
28
true
Original Function Parameters:
Component tFileInputXML
![clip_image009[1]](https://lh3.googleusercontent.com/-Vf6hjbQzWVo/VuCjOGdD28I/AAAAAAAAABg/5NGqvwOeb8E/clip_image009%25255B1%25255D_thumb.png?imgmax=800 "clip_image009[1]")
UNIQUE NAME
tFileInputXML_2
INPUT(S)
none
LABEL
AssetList
OUTPUT(S)
tXMLMap_1
Component Parameters:
Properties
Values
Unique Name
tFileInputXML_2
Component Name
tFileInputXML
Version
0.102 (ALPHA)
Family
| File/Input | XML |
Start
false
Startable
true
SUBTREE_START
false
END_OF_FLOW
false
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tFileInputXML
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH
false
!!!SCHEMA.NAME!!!
File name/Stream
“D:/Data/Qualys/Xml/tmpAssetList.xml”
Loop XPath query
“/HOST_LIST_OUTPUT/RESPONSE/HOST_LIST/HOST”
Mapping
[{QUERY=”ID”, NODECHECK=, SCHEMA_COLUMN=Qualys_Asset_ID}, {QUERY=”IP”, NODECHECK=, SCHEMA_COLUMN=IP}, {QUERY=”TRACKING_METHOD”, NODECHECK=, SCHEMA_COLUMN=TRACKING_METHOD}, {QUERY=”NETWORK_ID”, NODECHECK=, SCHEMA_COLUMN=NETWORK_ID}, {QUERY=”OS”, NODECHECK=, SCHEMA_COLUMN=OS}, {QUERY=”LAST_COMPLIANCE_SCAN_DATETIME”, NODECHECK=, SCHEMA_COLUMN=LAST_COMPLIANCE_SCAN_DATETIME}, {QUERY=”LAST_VULN_SCAN_DATETIME”, NODECHECK=, SCHEMA_COLUMN=LAST_VULN_SCAN_DATETIME}, {QUERY=”NETBIOS”, NODECHECK=, SCHEMA_COLUMN=NETBIOS}, {QUERY=”DNS”, NODECHECK=, SCHEMA_COLUMN=DNS}]
Limit
-1
Die on error
false
!!!SCHEMA_REJECT.NAME!!!
null:errorCode errorMessage
Advanced separator (for numbers)
false
Thousands separator
”,”
Decimal separator
”.”
Ignore the namespaces
false
Ignore DTD file
false
Generate a temporary file
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml”
Use Separator for mode Xerces
false
Field Separator
”,”
Generation mode
Dom4j
Validate date
false
Encoding
“UTF-8”
Min column number of optimize code
100
Label format
AssetList
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for metadata :
Column
Key
Type
Length
Precision
Nullable
Comment
Qualys_Asset_ID
false
Integer
9
true
IP
false
String
15
true
TRACKING_METHOD
false
String
11
true
NETWORK_ID
false
Integer
5
true
OS
false
String
80
true
LAST_COMPLIANCE_SCAN_DATETIME
false
java.util.Date
20
true
LAST_VULN_SCAN_DATETIME
false
java.util.Date
20
true
NETBIOS
false
String
15
true
DNS
false
String
45
true
Original Function Parameters:
Component tFileInputXML
![clip_image009[2]](https://lh3.googleusercontent.com/-kKGPEiqOTnU/VuCjOfMa5RI/AAAAAAAAABo/P-ubiMK0AVs/clip_image009%25255B2%25255D_thumb.png?imgmax=800 "clip_image009[2]")
UNIQUE NAME
tFileInputXML_5
INPUT(S)
none
LABEL
VulnerabilityList
OUTPUT(S)
tXMLMap_1
Component Parameters:
Properties
Values
Unique Name
tFileInputXML_5
Component Name
tFileInputXML
Version
0.102 (ALPHA)
Family
| File/Input | XML |
Start
false
Startable
true
SUBTREE_START
false
END_OF_FLOW
false
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tFileInputXML
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
REPOSITORY_ALLOW_AUTO_SWITCH
false
!!!SCHEMA.NAME!!!
File name/Stream
“D:/Data/Qualys/Xml/2tmpVulns.xml”
Loop XPath query
“/DATALIST/LIST/RECORD”
Mapping
[{QUERY=”KEY[@name=\“QID\”]”, NODECHECK=, SCHEMA_COLUMN=QID}, {QUERY=”KEY[@name=\“TITLE\”]”, NODECHECK=, SCHEMA_COLUMN=TITLE}, {QUERY=”KEY[@name=\“CATEGORY\”]”, NODECHECK=, SCHEMA_COLUMN=CATEGORY}, {QUERY=”KEY[@name=\“CVE_ID\”]”, NODECHECK=, SCHEMA_COLUMN=CVE_ID}, {QUERY=”KEY[@name=\“CVSS_BASE\”]”, NODECHECK=, SCHEMA_COLUMN=CVSS_BASE}, {QUERY=”KEY[@name=\“BUGTRAQ_ID\”]”, NODECHECK=, SCHEMA_COLUMN=BUGTRAQ_ID}, {QUERY=”KEY[@name=\“MODIFIED\”]”, NODECHECK=, SCHEMA_COLUMN=MODIFIED}, {QUERY=”KEY[@name=\“PUBLISHED\”]”, NODECHECK=, SCHEMA_COLUMN=PUBLISHED}]
Limit
-1
Die on error
false
!!!SCHEMA_REJECT.NAME!!!
null:errorCode errorMessage
Advanced separator (for numbers)
false
Thousands separator
”,”
Decimal separator
”.”
Ignore the namespaces
false
Ignore DTD file
false
Generate a temporary file
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace/temp.xml”
Use Separator for mode Xerces
false
Field Separator
”,”
Generation mode
Dom4j
Validate date
false
Encoding
“UTF-8”
Min column number of optimize code
100
Label format
VulnerabilityList
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for metadata :
Column
Key
Type
Length
Precision
Nullable
Comment
QID
false
Integer
6
true
TITLE
false
String
241
true
CATEGORY
false
String
27
true
CVE_ID
false
String
2855
true
CVSS_BASE
false
String
3
true
BUGTRAQ_ID
false
String
263
true
MODIFIED
false
String
28
true
PUBLISHED
false
String
28
true
Original Function Parameters:
Component tFileOutputDelimited
UNIQUE NAME
tFileOutputDelimited_2
INPUT(S)
tXMLMap_1, tFileInputXML_2, tFileInputXML_5, tFileInputDelimited_1
LABEL
__UNIQUE_NAME__
OUTPUT(S)
tSystem_2
Component Parameters:
Properties
Values
Unique Name
tFileOutputDelimited_2
Component Name
tFileOutputDelimited
Version
0.101 (ALPHA)
Family
File/Output
Startable
false
SUBTREE_START
false
END_OF_FLOW
true
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tFileOutputDelimited
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
!!!PROPERTY.NAME!!!
Use Output Stream
false
Output Stream
outputStream
File Name
“D:/Data/Qualys/VTM.csv”
Row Separator
”\r\n”
Use OS line separator as row separator when CSV Row Separator is set to CR,LF or CRLF.
true
CSV Row Separator
”\r\n”
Field Separator
”,”
Append
false
Include Header
true
Compress as zip file
false
REPOSITORY_ALLOW_AUTO_SWITCH
false
Schema
Advanced separator (for numbers)
false
Thousands separator
”,”
Decimal separator
”.”
CSV options
true
Escape char
”””
Text enclosure
”””
Create directory if does not exist
true
Split output in several files
false
Rows in each output file
1000
Custom the flush buffer size
false
Row number
1
Output in row mode
false
Encoding
“ISO-8859-15”
Don’t generate empty file
false
Min column number of optimize code
90
Label format
__UNIQUE_NAME__
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for VTM :
Column
Key
Type
Length
Precision
Nullable
Comment
QUALYS_TICKET_NUMBER
false
Integer
10
true
TICKET_CREATION_DATETIME
false
String
20
true
TICKET_RESOLVED_DATETIME
false
String
28
true
TICKET_MODIFIED_DATETIME
false
String
28
true
TICKET_DUE_DATETIME
false
String
20
true
TICKET_CURRENT_STATE
false
String
6
true
IP
false
String
15
true
PORT
false
String
4
true
QUALYS_SEVERITY
false
String
255
true
QID
false
Integer
6
true
TRACKING_METHOD
false
String
11
true
OS
false
String
80
true
NETBIOS
false
String
15
true
DNS
false
String
45
true
LAST_COMPLIANCE_SCAN_DATETIME
false
java.util.Date
20
true
LAST_VULN_SCAN_DATETIME
false
java.util.Date
20
true
TITLE
false
String
241
true
CATEGORY
false
String
27
true
CVE_ID
false
String
2855
true
CVSS_BASE
false
String
3
true
BUGTRAQ_ID
false
String
263
true
QID_MODIFIED
false
String
28
true
QID_PUBLISHED
false
String
28
true
CIDR
false
String
14
true
Zone
false
String
8
true
Department
false
String
3
true
Function
false
String
15
true
Original Function Parameters:
Component tHttpRequest
UNIQUE NAME
tHttpRequest_2
INPUT(S)
tSystem_1
LABEL
__UNIQUE_NAME__
OUTPUT(S)
tFileInputXML_1
Component Parameters:
Properties
Values
Unique Name
tHttpRequest_2
Component Name
tHttpRequest
Version
0.101 (ALPHA)
Family
Internet
Start
false
Startable
true
SUBTREE_START
true
END_OF_FLOW
true
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tHttpRequest
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
REPOSITORY_ALLOW_AUTO_SWITCH
false
Property
null:ResponseContent
URI
“https://qualysapi.qualys.eu/api/2.0/fo/asset/host/?action=list&truncation_limit=1000000&details=All&vm_scan_since=2015-01-01”
Method
POST
Post parameters from file
Write response content to file
true
“D:/Data/Qualys/Xml/tmpAssetList.xml”
Create directory if not exists
false
Headers
[{HEADER_NAME=”X-Requested-With”, HEADER_VALUE=”Talend”}, {HEADER_NAME=”Authorization”, HEADER_VALUE=”Basic Z29kbW9kZTp1ODAwOHBhc3N3ZG5vdGxpa2VseXRvYmVoZXJlLi4uLi4xMjM=”}]
Need authentication
false
user
””
password
**
Die on error
false
Label format
__UNIQUE_NAME__
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for tHttpRequest_2 :
Column
Key
Type
Length
Precision
Nullable
Comment
ResponseContent
false
String
true
Original Function Parameters:
Component tSystem
UNIQUE NAME
tSystem_1
INPUT(S)
none
LABEL
__UNIQUE_NAME__
OUTPUT(S)
tHttpRequest_2
Component Parameters:
Properties
Values
Unique Name
tSystem_1
Component Name
tSystem
Version
0.101 (ALPHA)
Family
System
Start
true
Startable
true
SUBTREE_START
true
END_OF_FLOW
true
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tSystem
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
Use Home Directory
false
Home Directory
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace”
Use Single Command
true
Command
“C:\\Python27\\python.exe D:\\Data\\Qualys\\ExternalJob\\QualysDownloadSelenium.py”
Use Array Command
false
Command
[]
Standard Output
OUTPUT_TO_CONSOLE
Error Output
OUTPUT_TO_CONSOLE
REPOSITORY_ALLOW_AUTO_SWITCH
false
!!!SCHEMA.NAME!!!
null:
Environment variables
[]
Label format
__UNIQUE_NAME__
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for tSystem_1 :
Column
Key
Type
Length
Precision
Nullable
Comment
Original Function Parameters:
Component tSystem
![clip_image015[1]](https://lh3.googleusercontent.com/-foV_MLovWvk/VuCjP17BgmI/AAAAAAAAAB4/80UeYBZm56E/clip_image015%25255B1%25255D_thumb.png?imgmax=800 "clip_image015[1]")
UNIQUE NAME
tSystem_2
INPUT(S)
tFileOutputDelimited_2
LABEL
__UNIQUE_NAME__
OUTPUT(S)
none
Component Parameters:
Properties
Values
Unique Name
tSystem_2
Component Name
tSystem
Version
0.101 (ALPHA)
Family
System
Start
false
Startable
true
SUBTREE_START
true
END_OF_FLOW
true
Activate
true
DUMMY
false
tStatCatcher Statistics
false
Help
org.talend.help.tSystem
Update components
true
IREPORT_PATH
JAVA_LIBRARY_PATH
C:\install\TOS_BD-20151214_1327-V6.1.1\configuration\lib\java
Subjob color
Title color
Use Home Directory
false
Home Directory
“C:/install/TOS_BD-20151214_1327-V6.1.1/workspace”
Use Single Command
true
Command
“cmd /c del D:\\Data\\Qualys\\Xml\\*.xml”
Use Array Command
false
Command
[]
Standard Output
OUTPUT_TO_CONSOLE
Error Output
OUTPUT_TO_CONSOLE
REPOSITORY_ALLOW_AUTO_SWITCH
false
!!!SCHEMA.NAME!!!
null:
Environment variables
[]
Label format
__UNIQUE_NAME__
Hint format
__UNIQUE_NAME__
__COMMENT__
Connection format
row
Show Information
false
Comment
Use an existing validation rule
false
Validation Rule Type
Schema for VTM :
Column
Key
Type
Length
Precision
Nullable
Comment
QUALYS_TICKET_NUMBER
false
Integer
10
true
TICKET_CREATION_DATETIME
false
String
20
true
TICKET_RESOLVED_DATETIME
false
String
28
true
TICKET_MODIFIED_DATETIME
false
String
28
true
TICKET_DUE_DATETIME
false
String
20
true
TICKET_CURRENT_STATE
false
String
6
true
IP
false
String
15
true
PORT
false
String
4
true
QUALYS_SEVERITY
false
String
255
true
QID
false
Integer
6
true
TRACKING_METHOD
false
String
11
true
OS
false
String
80
true
NETBIOS
false
String
15
true
DNS
false
String
45
true
LAST_COMPLIANCE_SCAN_DATETIME
false
java.util.Date
20
true
LAST_VULN_SCAN_DATETIME
false
java.util.Date
20
true
TITLE
false
String
241
true
CATEGORY
false
String
27
true
CVE_ID
false
String
2855
true
CVSS_BASE
false
String
3
true
BUGTRAQ_ID
false
String
263
true
QID_MODIFIED
false
String
28
true
QID_PUBLISHED
false
String
28
true
CIDR
false
String
14
true
Zone
false
String
8
true
Department
false
String
3
true
Function
false
String
15
true
Original Function Parameters:
Component tXMLMap
UNIQUE NAME
tXMLMap_1
INPUT(S)
tXMLMap_1, tFileInputXML_2, tFileInputXML_5, tFileInputDelimited_1
LABEL
__UNIQUE_NAME__
OUTPUT(S)
tFileOutputDelimited_2
Component Parameters:
Properties
Values
Activate
true
tStatCatcher Statistics
false
Map Editor:
Keep order for document
false
Show Information
false
Comment
Use an existing validation rule
false
Schema for VTM :
Column
Key
Type
Length
Precision
Nullable
Comment
QUALYS_TICKET_NUMBER
false
Integer
10
true
TICKET_CREATION_DATETIME
false
String
20
true
TICKET_RESOLVED_DATETIME
false
String
28
true
TICKET_MODIFIED_DATETIME
false
String
28
true
TICKET_DUE_DATETIME
false
String
20
true
TICKET_CURRENT_STATE
false
String
6
true
IP
false
String
15
true
PORT
false
String
4
true
QUALYS_SEVERITY
false
String
255
true
QID
false
Integer
6
true
TRACKING_METHOD
false
String
11
true
OS
false
String
80
true
NETBIOS
false
String
15
true
DNS
false
String
45
true
LAST_COMPLIANCE_SCAN_DATETIME
false
java.util.Date
20
true
LAST_VULN_SCAN_DATETIME
false
java.util.Date
20
true
TITLE
false
String
241
true
CATEGORY
false
String
27
true
CVE_ID
false
String
2855
true
CVSS_BASE
false
String
3
true
BUGTRAQ_ID
false
String
263
true
QID_MODIFIED
false
String
28
true
QID_PUBLISHED
false
String
28
true
CIDR
false
String
14
true
Zone
false
String
8
true
Department
false
String
3
true
Function
false
String
15
true
P.S. for the astute, no I did not leave any credentials of use :) it’s garbage…