summaryrefslogtreecommitdiffstats
path: root/meta/recipes-core/meta/cve-update-db-native.bb
Commit message (Collapse)AuthorAgeFilesLines
* cve-update-db-native: removeRoss Burton2023-06-271-288/+0
| | | | | | | | | | | The replacement cve-update-nvd2-native is working, so we can remove the old recipe now. (From OE-Core rev: 17eb07885147a06bf7bdd43f7869fe4411be80a0) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* Revert "cve-update-db-native: show IP on failure"Ross Burton2023-01-291-4/+1
| | | | | | | | | | | | The bug this was intended to find has been resolved, so we don't need this extra noise in the logs. This reverts commit bd1edfa3d5b1c24cd1192620f7085ee1df03e1af. (From OE-Core rev: 54ae53ef3e4ad51fc9a01155588b62f29c994742) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: show IP on failureRoss Burton2023-01-141-1/+4
| | | | | | | | | | | | | | We get random SSL failures when fetching the CVE database, and it's notable that the NVD server is behind a DNS round-robin or geographically diverse servers. On a hunch that there is one misconfigured server, dump the IP that we connected to. (From OE-Core rev: 91f46d431dc8f40e8c6475c800bb61cb08b82b0a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: avoid incomplete updatesMarta Rybczynska2023-01-061-22/+61
| | | | | | | | | | | | | | | | | | The database update has been done on the original file. In case of network connection issues, temporary outage of the NVD server or a similar situation, the function could exit with incomplete data in the database. This patch solves the issue by performing the update on a copy of the database. It replaces the main one only if the whole update was successful. See https://bugzilla.yoctoproject.org/show_bug.cgi?id=14929 Reported-by: Alberto Pianon <alberto@pianon.eu> (From OE-Core rev: 8efe99214d8b005f0ecac690ce5ba17b31758f92) Signed-off-by: Marta Rybczynska <marta.rybczynska@linaro.org> Signed-off-by: Alexandre Belloni <alexandre.belloni@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: add timeout to urlopen() callsFrank de Brabander2022-10-251-2/+7
| | | | | | | | | | | | | | | The urlopen() call can block indefinitely under some circumstances. This can result in the bitbake process to run endlessly because of the 'do_fetch' task of cve-update-bb-native to remain active. This adds a default timeout of 60 seconds to avoid this hang, while being large enough to minimize the risk of unwanted timeouts. (From OE-Core rev: e5f6652854f544106b40d860de2946954de642f3) Signed-off-by: Frank de Brabander <debrabander@gmail.com> Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: close cursors as soon as possibleRoss Burton2022-09-021-23/+28
| | | | | | | | | | | | | | We can have multiple processes reading the database at the same time, and cursors only release their locks when they're garbage collected. This might be the cause of random sqlite errors on the autobuilder, so explicitly close the cursors when we're done with them. (From OE-Core rev: 5d2e90e4a58217a943ec21140bc2ecdd4357a98a) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: make it possible to disable database updatesMarta Rybczynska2022-06-061-2/+4
| | | | | | | | | | | | | | | Make it possible to disable the database update completely by using a negative update interval CVE_DB_UPDATE_INTERVAL. Disabling the update is useful when running multiple parallel builds when we want to have a control on the database version. This allows coherent cve-check results without an database update for only some of the builds. (From OE-Core rev: b5c2269240327c2a8f93b9e55354698f52c976f3) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: let the user to drive the update intervalMarta Rybczynska2022-05-031-1/+9
| | | | | | | | | | | | | Add a new variable CVE_DB_UPDATE_INTERVAL allowing the user to set the database update interval. - a positive value sets an interval (in seconds) - a zero ("0") forces the database update (From OE-Core rev: fe7bc6f16184d5ebdb1dd914b6dcb75c9e5e0c9c) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: update the CVE database once a day onlyMarta Rybczynska2022-05-031-2/+2
| | | | | | | | | | | | | | | | The update of the NVD database was expected to happen once per hour. However, the database file date changes only if the content was actually updated. In practice, the check worked for the first hour after the new download. As the NVD database changes usually only once a day, we can just update it less frequently. (From OE-Core rev: 35bccdedadeaba820d58b69fe74ce5e4c1f577e3) Signed-off-by: Marta Rybczynska <marta.rybczynska@huawei.com> Signed-off-by: Luca Ceresoli <luca.ceresoli@bootlin.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: skip on empty cpe23UriKonrad Weihmann2021-04-231-1/+6
| | | | | | | | | | | | | | Recently an entry in the NVD DB appeared that looks like that {'vulnerable': True, 'cpe_name': []}. As besides all the vulnerable flag no data is present we would get a KeyError exception on acccess. Use get method on dictionary and return if no meta data is present Also quit if the length of the array after splitting is less than 6 (From OE-Core rev: 00ce2796d97de2bc376b038d0ea7969088791d34) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: Allow to overrule the URL in a bbappend.jan2021-03-311-2/+3
| | | | | | | | | | With this small patch, it's possible to overrule the public URL with a local mirror for those without Internet access. (From OE-Core rev: 2d903126e8bbece3a5171c3488c3deae1f0aa3ee) Signed-off-by: Jan Vermaete <jan.vermaete@gmail.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: consider version suffix when update CVE dbLee Chee Yang2021-03-201-1/+6
| | | | | | | | | | | | | | | some record from NVD can merge or split suffix from version, for example: CVE-2017-15906 "cpe23Uri" : "cpe:2.3:a:openbsd:openssh:5.0:p1:*:*:*:*:*:*" "cpe23Uri" : "cpe:2.3:a:openbsd:openssh:4.7p1:*:*:*:*:*:*:*" in such case include the suffix into version when update local CVE db. (From OE-Core rev: 13cc68197f81bb7c76fa1abecc5dd720b8bdb8d5) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: handle all-wildcard versionsRoss Burton2020-11-291-1/+6
| | | | | | | | | | | If a CPE version field is just *:*:*:* it should be handled the same as -:*:*:*, that is 'all versions'. To ease handling, transform this case to use -. (From OE-Core rev: 04a9bc4ca5294fe6834513669c7746a824d12b04) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: remove unused variableChris Laplante2020-09-151-1/+0
| | | | | | | (From OE-Core rev: 790f8a5bfbd1851226fa5fc5b09e79f6396e747d) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: move -journal checking into do_fetchChris Laplante2020-09-151-6/+8
| | | | | | | | | | | It was always questionable to do this in an anonymous function, but now with multiconfig it is a critical mistake and leads to more strange "Exception: sqlite3.OperationalError: disk I/O error" errors. (From OE-Core rev: fcbc9d0e8421a3a559db05005ba042337cd5d864) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: be less magical about checking whether the cve-check ↵Chris Laplante2020-09-151-2/+1
| | | | | | | | | class is enabled (From OE-Core rev: cbc717350b21fb417fa6f83621022ac633a26124) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: use fetch taskRoss Burton2020-09-121-4/+5
| | | | | | | | | | Instead of inventing a new task to fetch the CVE data, use the existing fetch task. (From OE-Core rev: f5f97d33a1703d75b9fd9760f2c7767081538e00) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: add more logging when fetchingRoss Burton2020-09-121-1/+5
| | | | | | | | | Add some debug logging when fetching the CVE data. (From OE-Core rev: 9b230584664873af2ab453b8153b1ad276d3b0af) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: remove unused importRoss Burton2020-09-121-1/+1
| | | | | | | | | This import isn't used anymore, so remove it. (From OE-Core rev: d4f0e43bdf1de972e95869f033dd45f8d420c400) Signed-off-by: Ross Burton <ross.burton@arm.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: use context manager for cve_fChris Laplante2020-09-101-4/+1
| | | | | | (From OE-Core rev: ff422652e1b5db62205fafc75ce56bb5951d478d) Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check/cve-update-db-native: use lockfile to fix usage under multiconfigChris Laplante2020-09-101-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously CVE_CHECK_DB_FILE / CVE_CHECK_DB_DIR was the same across multiconfigs which led to a race condition wherein multiple cve-update-db-native:do_populate_cve_db tasks could attempt to write to the same sqlite database. This led to the following task failure: Error executing a python function in exec_python_func() autogenerated: The stack trace of python calls that resulted in this exception/failure was: File: 'exec_python_func() autogenerated', lineno: 2, function: <module> 0001: *** 0002:do_populate_cve_db(d) 0003: File: '/mnt/data/agent/work/74f119cccb44f133/yocto/sources/poky/meta/recipes-core/meta/cve-update-db-native.bb', lineno: 103, function: do_populate_cve_db 0099: if year == date.today().year: 0100: cve_f.write('CVE database update : %s\n\n' % date.today()) 0101: 0102: cve_f.close() *** 0103: conn.commit() 0104: conn.close() 0105:} 0106: 0107:def initialize_db(c): Exception: sqlite3.OperationalError: disk I/O error Use a lockfile to ensure multiple tasks don't step over each other. (From OE-Core rev: 24e9380643a2ae3fcae193519cb64aedaf682153) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: add progress handlerChris Laplante2020-09-101-43/+47
| | | | | | | (From OE-Core rev: 79ae2e82b8ec11578177f428060b568d6c7d44ca) Signed-off-by: Chris Laplante <chris.laplante@agilent.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update: handle baseMetricV2 as optionalKonrad Weihmann2020-07-271-4/+9
| | | | | | | | | | | Currently in NVD DB an item popped up, which hasn't set baseMetricV2. Let the parser handle it as an optional item. In case use baseMetricV2 before baseMetricV3 (From OE-Core rev: fdcbf3f28289188c5a97664d1421d4a5c4991eda) Signed-off-by: Konrad Weihmann <kweihmann@outlook.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: fix DB file version for '-'Lee Chee Yang2020-04-051-0/+3
| | | | | | | | | | | fix logic for CVE DB update so that when the CPE version is '-', it keeps the version as '-' in the DB file too and leave other operation as blank. (From OE-Core rev: 0ec35b00d65c70485eb1efe602d9fca53e926811) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: CPE version '-' as all versionLee Chee Yang2020-04-011-1/+1
| | | | | | | | | | | | | | | | | | CPE version could be '-' to mean no version info. Current cve_check treat it as not valid and does not report these CVE but some of these could be a valid vulnerabilities. Since non-valid CVE can be whitelisted, so treat '-' as all version and report all these CVE to capture possible vulnerabilities. Non-valid CVE to be whitelisted separately. [YOCTO #13617] (From OE-Core rev: c69ee3594079589d27c10db32bc288566ebde9ef) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: clean DB if temporary file existLee Chee Yang2020-03-291-1/+8
| | | | | | | | | | | | | | | | when do_populate_cve_db forced stop at certain point, the DB execution are stoped however the temporary database file (DB-JOURNAL) are not removed. This db-journal file indicates that DB is incomplete and set DB in readonly mode. So when db-journal exist, remove both DB and the db-journal and build the DB again from scratch. [YOCTO #13682] (From OE-Core rev: 70713df25b8ae27a21e53b0b9234567d7053800a) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: prevent fatal error cause by networkLee Chee Yang2020-03-291-1/+7
| | | | | | | | | | | prevent cve-check from fatal error cause by network issue. [YOCTO #13680] (From OE-Core rev: da358e415726f836e6b3e69ad7fbeffb50bd402d) Signed-off-by: Lee Chee Yang <chee.yang.lee@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: Switch to NVD CVE JSON feed version 1.1Niko Mauno2019-12-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Switch to recently released version 1.1 of NVD CVE JSON feed, as in https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release it is mentioned that Due to changes required to support CVSS v3.1 scoring, the JSON vulnerability feeds must be modified. This will require the consumers of this data to update their internal processes. We will be providing the JSON 1.1 schema on the data feeds page and the information below to prepare for this transition. ... The JSON 1.1 data feeds will be available on September 9th, 2019. At that time the current JSON 1.0 data feeds will no longer available. This change was tested briefly by issuing 'bitbake core-image-minimal' with 'cve-check.bbclass' inherited via local.conf, and then comparing the content between the resulting two 'DEPLOY_DIR_IMAGE/core-image-minimal-qemux86.cve' files, which did not seem to contain any other change, except total of 167 entries like CVSS v3 BASE SCORE: 0.0 were replaced with similar 'CVSS v3 BASE SCORE:' entries which had scores that were greater than '0.0' (up to '9.8'). (From OE-Core rev: cc20e4d8ff2f3aa52a2658404af9a0ff358cc323) Signed-off-by: Niko Mauno <niko.mauno@iki.fi> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: clean up proxy handlingRoss Burton2019-11-211-26/+5
| | | | | | | | | | urllib handles adding proxy handlers if the proxies are set in the environment, so call bb.utils.export_proxies() to do that and remove the manual setup. (From OE-Core rev: 6b73004668b3b71c9c38814b79fbb58c893ed434) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: add an index on the CVE ID columnRoss Burton2019-11-211-0/+3
| | | | | | | | | | Create an index on the PRODUCTS table which contains a row for each CPE, drastically increasing the performance of lookups for a specific CVE. (From OE-Core rev: b4048b05b3a00d85c40d09961f846eadcebd812e) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: don't hardcode the database nameRoss Burton2019-11-211-2/+2
| | | | | | | | | | Don't hardcode the database filename, there's a variable for this in cve-check.bbclass. (From OE-Core rev: 0d188a9dc4ae64c64cd661e9d9c3841e86f226ab) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: don't refresh more than once an hourRoss Burton2019-11-141-1/+9
| | | | | | | | | | | We already fetch the yearly CVE metadata and check that for updates before downloading the full data, but we can speed up CVE checking further by only checking the CVE metadata once an hour. (From OE-Core rev: 50d898fd360c58fe85460517d965f62b7654771a) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: fix https proxy issuesChin Huat Ang2019-07-261-11/+30
| | | | | | | | | | | | | | When https_proxy is set, use proxy opener to open CVE metadata and database URLs, otherwise fallback to the urllib.request.urlopen. Also fix a minor issue where the json database which has been gzip decompressed as byte object should be decoded as utf-8 string as expected by update_db. (From OE-Core rev: 95438d52b732bec217301fbfc2fb019bbc3707c8) Signed-off-by: Chin Huat Ang <chin.huat.ang@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: clean up JSON fetchingRoss Burton2019-07-221-17/+12
| | | | | | | | | | | | | Currently the code fetches the compressed JSON, writes it to a temporary file, uncompresses that with gzip and passes the fake file object to update_db(). Instead, uncompress the gzip'd data in memory and pass the JSON directly to update_db(). (From OE-Core rev: 9422745979256c442f533770203f62ec071c18fb) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: improve metadata parsingRoss Burton2019-07-221-8/+10
| | | | | | | | | | | | | | | | The metadata parser is fragile: first it coerces a bytes() to a str() (so the string is b'LastModifiedDate:2019...'), assumes the first line is the date, and then uses a regex to parse (which then includes the trailing quote as part of the date). Clean this up by parsing the bytes as UTF-8 (ASCII is probably fine, but this is safer), iterate through the lines and split on colons to find the right key/value pair. (From OE-Core rev: bb4e53af33d6ca1e9346464adbdc1b39c47530f3) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: use executemany() to optimise CPE insertionRoss Burton2019-07-221-53/+32
| | | | | | | | | | Instead of calling execute() repeatedly, rewrite the function to be a generator and use executemany() for performance. (From OE-Core rev: b309840b6aa3423b909a43499356e929c8761318) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db: actually inherit nativeRoss Burton2019-07-191-2/+1
| | | | | | | | | The recipe was called -native but didn't inherit native. (From OE-Core rev: f0d822fad2a163d1ee32ed3b4c0359245140e19b) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: use os.path.join instead of +Ross Burton2019-07-191-4/+4
| | | | | | | (From OE-Core rev: 4b301030cf9cf7a981dcff85a50e915c045e3130) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: Remove hash column from database.Pierre Le Magourou2019-07-191-14/+7
| | | | | | | | | | | | djb2 hash algorithm was found to do collisions, so the database was sometime missing data. Remove this hash mechanism, clear and populate elements from scratch in PRODUCTS table if the current year needs an update. (From OE-Core rev: 78de2cb39d74b030cd4ec811bf6f9a6daa003d19) Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: use SQL placeholders instead of format stringsRoss Burton2019-07-181-1/+1
| | | | | | | (From OE-Core rev: 91770338f76ef35f3c4eeac216eb9d2b3188e575) Signed-off-by: Ross Burton <ross.burton@intel.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db-native: Skip recipe when cve-check class is not loaded.Pierre Le Magourou2019-07-101-0/+5
| | | | | | | (From OE-Core rev: c807c2a6409e122599196cd914a638b00121cab6) Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-update-db: Use NVD CPE data to populate PRODUCTS tablePierre Le Magourou2019-07-091-14/+74
| | | | | | | | | | | | | Instead of using expanded list of affected versions that is not reliable, use the 'cpe_match' node in the 'configurations' json node. For cve-check to correctly match affected CVE, the sqlite database need to contain operator_start, operator_end and the corresponding versions fields. (From OE-Core rev: f7676e9a38d595564922e5f59acbc69c2109a78f) Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
* cve-check: Depends on cve-update-db-nativePierre Le Magourou2019-07-091-0/+143
do_populate_cve_db is a native task. (From OE-Core rev: 4078da92b49946848cddebe1735f301af161e162) Signed-off-by: Pierre Le Magourou <pierre.lemagourou@softbankrobotics.com> Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
generated by cgit v1.2.3-54-g00ecf (git 2.39.0) at 2024-05-12 23:19:39 +0000