Skip to content

added validation for CFD CSCwp64296#307

Open
psureshb wants to merge 14 commits intodatacenter:masterfrom
psureshb:psureshb
Open

added validation for CFD CSCwp64296#307
psureshb wants to merge 14 commits intodatacenter:masterfrom
psureshb:psureshb

Conversation

@psureshb
Copy link

No description provided.

@psureshb
Copy link
Author

pytest logs
###################################
===================================================================== test session starts ======================================================================
platform linux -- Python 3.8.10, pytest-7.4.0, pluggy-1.2.0 -- /data/ssd/psureshb/repo/pytest_env/bin/python3
cachedir: .pytest_cache
rootdir: /data/ssd/psureshb/repo/preCFD/ACI-Pre-Upgrade-Validation-Script
configfile: pytest.ini
collected 13 items

tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_non_affected_tversion_affected_cversion_no_exception_MACs]
------------------------------------------------------------------------ live log setup ------------------------------------------------------------------------
[10:45:39.988 INFO init_system:6066(MainThread)] Cleaning up previous run files in preupgrade_validator_logs/
[10:45:39.989 INFO init_system:6068(MainThread)] Creating directories preupgrade_validator_logs/ and preupgrade_validator_logs/json_results/
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:39.990 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:39.991 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:39.991 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:39.992 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 7%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_non_affected_cversion_affected_tversion_no_exception_MACs]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.003 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.004 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.004 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.005 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 15%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_non_affected_cversion_tversion_no_exception_MACs]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.016 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.017 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.018 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.019 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 23%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_non_affected_cversion_tversion_with_exception_MACs]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.032 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.032 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.033 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.033 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 30%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_affected_edge_cversion_tversion_no_exception_MACs]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.046 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.047 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.048 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.048 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 38%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_affected_cversion_tversion_no_exception_MACs]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.061 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.062 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.063 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.063 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 46%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_affected_cversion_tversion_exception_MACs_with_exceptcont_listeners]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.076 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.077 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.077 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.078 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 53%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[PASS_affected_edge_cversion_tversion_exception_MACs_with_exceptcont_listeners]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.092 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.093 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.093 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.094 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 61%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[MANUAL_tversion_not_provided]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.107 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.108 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.108 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check| 0/1 checks completed
[10:45:40.109 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 69%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[FAIL_affected_edge_cversion_tversion_exception_MACs_one_missing_exceptcont_listener]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.122 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.122 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.123 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check| 0/1 checks completed
[10:45:40.124 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 76%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[FAIL_affected_cversion_tversion_exception_MACs_31_missing_exceptcont_listeners]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.138 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.139 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.139 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check| 0/1 checks completed
[10:45:40.140 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 84%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[FAIL_affected_cversion_tversion_exception_MACs_many_missing_exceptcont_listeners]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.153 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.153 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.154 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check| 0/1 checks completed
[10:45:40.155 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [ 92%]
tests/checks/rogue_ep_coop_exception_mac_check/test_rogue_ep_coop_exception_mac_check.py::test_rogue_ep_coop_exception_mac_check[FAIL_affected_cversion_tversion_exception_MACs_no_exceptcont_listeners]
------------------------------------------------------------------------ live log call -------------------------------------------------------------------------
[10:45:40.167 INFO init_result:1182(MainThread)] Initialized in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
[10:45:40.168 INFO _start_thread:1105(MainThread)] (rogue_ep_coop_exception_mac_check) Starting thread.
[10:45:40.168 INFO wrapper:1349(rogue_ep_coop_exception_mac_check)] Start rogue_ep_coop_exception_mac_check
[10:45:40.169 INFO update_result:1191(rogue_ep_coop_exception_mac_check)] Finalized result in preupgrade_validator_logs/json_results/rogue_ep_coop_exception_mac_check.json
PASSED [100%]

====================================================================== 13 passed in 0.20s ======================================================================

Pre-upgrade validation script logs:---------->
######################################################
==== 2025-12-24T10-42-44+0000, Script Version v4.0.1 ====

!!!! Check https://github.com/datacenter/ACI-Pre-Upgrade-Validation-Script for Latest Release !!!!

To use a non-default Login Domain, enter apic#DOMAIN\USERNAME
Gathering Node Information...

Current APIC Version...6.1(5e)
Lowest Switch Version...6.1(5e)

Target APIC version is overridden to 6.1(1f)

Collecting VPC Node IDs...

Progress: |----------------------------------------------------------------------------------------------------| 0/1 checks completed
Progress: |████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 checks completed

=== Check Result (failed only) ===

=== Summary Result ===

PASS : 1
FAIL - OUTAGE WARNING!! : 0
FAIL - UPGRADE FAILURE!! : 0
MANUAL CHECK REQUIRED : 0
POST UPGRADE CHECK REQUIRED : 0
N/A : 0
ERROR !! : 0
TOTAL : 1
preupgrade_validator_logs/summary.json0000644003601600360160000000027515122741651017655 0ustar adminadmin{
"PASS": 1,
"FAIL - OUTAGE WARNING!!": 0,
"FAIL - UPGRADE FAILURE!!": 0,
"MANUAL CHECK REQUIRED": 0,
"POST UPGRADE CHECK REQUIRED": 0,
"N/A": 0,
"ERROR !!": 0,
"TOTAL": 1

@diwakarmishra1190
Copy link

Let's test the actual fail scenario using below method.

  1. Login as root
  2. enable testapi
  3. find any preListener exception list MO dn (random)
  4. using icurl set status delete
  5. check the count of total dn.
  6. if it is less than 32 now run your script and see if it is detecting the failure or not.
  7. once test is done set the dn status to created.

Example:

bdsol-aci01-apic1#
bdsol-aci01-apic1# acidiag dbgtoken
0HDIBKCZXEQI

bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1# ssh root@0

root@0.0.0.0's password:
root@bdsol-aci01-apic1:#
root@bdsol-aci01-apic1:
# enable_testapi.bin 3600
Decoded seconds argument 3600 to be 3600
PAM_SERVER UDS SOCKET PATH = /var/run/mgmt/socket/pam
Connection successful - attempting to send testapi enable directive
Sent testapi enable directive (total pkt len 8)
Received response from testapi server
testapi enable directive acknowledged by REST endpoint
Success : testapi functionality is now enabled for 3600 seconds
root@bdsol-aci01-apic1:#
root@bdsol-aci01-apic1:
# exit
logout
Connection to 0.0.0.0 closed.
bdsol-aci01-apic1# icurl -kX POST -g http://localhost:7777/testapi/node/mo/.xml -d ""

bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1# moquery -c presListener -f 'pres.Listener.dn*"exceptcont"' | grep dn | grep resregistry-32
bdsol-aci01-apic1# icurl -kX POST -g http://localhost:7777/testapi/node/mo/.xml -d ""

bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1# moquery -c presListener -f 'pres.Listener.dn*"exceptcont"' | grep dn | grep resregistry-32
dn : resregistry/resregistry-32/relnholder/rspresClass-[resregistry/resregistry-32/class-19076]/list-[exceptcont]
bdsol-aci01-apic1#
bdsol-aci01-apic1#
bdsol-aci01-apic1# icurl -kX POST -g http://localhost:7777/testapi/node/mo/.xml -d ""

bdsol-aci01-apic1# moquery -c presListener -f 'pres.Listener.dn*"exceptcont"' | grep dn | wc -l
28 <<<<<<<<<<<<
bdsol-aci01-apic1#
bdsol-aci01-apic1# icurl -kX POST -g http://localhost:7777/testapi/node/mo/.xml -d ""

bdsol-aci01-apic1# moquery -c presListener -f 'pres.Listener.dn*"exceptcont"' | grep dn | wc -l
29 <<<<<<<<<<<<<
bdsol-aci01-apic1#

@psureshb
Copy link
Author

rogue_ep_coop_exception_mac_check_Fail.txt
This file contains the failure logs.

Copy link

@diwakarmishra1190 diwakarmishra1190 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
tested Locally

Copy link

@diwakarmishra1190 diwakarmishra1190 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

New changes are working fine tested locally.

Copy link
Collaborator

@takishida takishida left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@psureshb @diwakarmishra1190

I've updated the doc with the information that is relevant to users along with the new conditions.

Correct me if I'm wrong, but I doubt that the current condition checked by this rule will ever hit because I believe presListener will be missing on APICs after APICs are upgraded. But you are checking the current version being the older one and the target version being the affected one. That is only the case BEFORE APICs (or anything) are upgraded. In other words, the presListener issue has yet to happen.

Or if presLIsteners are created on APICs when switches are upgraded instead of APICs, using that as a condition doesn't make sense at all. This script is to prevent the issue from happening by detecting the conditions before the issue happens instead of checking the symptoms of the issue after hitting it.

Can you please review the updated doc and update the rule logic accordingly?

# if cversion.newer_than("3.1(2v)") and tversion.older_than("6.1(3g)"):
if (
(cversion.same_as("5.2(3e)") or cversion.newer_than("5.2(3e)")) and
(cversion.same_as("6.0(3g)") or cversion.older_than("6.0(3g)")) and
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why 6.0(3g)? The enhancement that caused CSCwp64296 was introduced in the first release of 6.0(3) which is 6.0(3d).

)
):
# endpoint to fetch the rogue exception MACs
exception_mac_api = 'fvRogueExceptionMac.json?query-target-filter=and(wcard(fvRogueExceptionMac.dn,"([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}"))'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you matching MAC address in regex here? fvRogueExceptionMac is for MAC as the name suggests. Its DN always has a MAC address.

Unless I'm missing something, this is just causing unnecessary query work on the APICs.

exception_mac_api = 'fvRogueExceptionMac.json?query-target-filter=and(wcard(fvRogueExceptionMac.dn,"([0-9a-fA-F]{2}:){5}[0-9a-fA-F]{2}"))'

# endpoint to fetch the presListener entries
presListener_api = 'presListener.json?query-target-filter=and(wcard(presListener.dn,"exceptcont"))'
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of regex(wcard)), you should do exact match with lstDn such as eq(presListener.lstDn,"exceptcont")

And use rsp-subtree-include=count instead of having APICs return all of the contents because what we need to know is only the number of objects.

exception_macs = icurl('class', exception_mac_api)

if exception_macs:
prints("Found {} exception MACs, checking presListener entries...".format(len(exception_macs)))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use prints() inside a check function.

In the new framework with the progress bar, it will end up like this, breaking the progress bar view. Please use logging instead, if needed. Otherwise, those info should be stored in the data so that it can be displayed in the result table and JSON file.

--- skip ---
Collecting VPC Node IDs...

Found 1 exception MACs, checking presListener entries...-------------------------------------------------------| 0/1 checks completed     <--- !!!!!
Progress: |████████████████████████████████████████████████████████████████████████████████████████████████████| 1/1 checks completed


=== Check Result (failed only) ===


# Target version check
if not tversion:
prints("Target version not provided, skipping check.")
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
prints("Target version not provided, skipping check.")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants