BAD GPU retirement policy by total bad pages
Subsystems (2)
cover BAD GPU retirement policy by total bad pages
The series is to enable the feature of GPU RMA(Return Merchandise Authorization) which is trigged when bad pages detected by RAS ECC exceed the threshold value.
When the saved bad pages written to eeprom reach the threshold, one ras recovery will be issued immediately and the recovery will fail to tell user that the GPU is BAD and needs to be retired for further check.
During bootup, similar BAD GPU check is conducted as well when eeprom get initialized, and it will break boot up for user's awareness.
User could set bad_page_threshold=0 when probing amdgpu driver to disable this feature to bring up GPU, and reset eeprom later.
Guchun Chen (5):
drm/amdgpu: add bad page count threshold in module parameter drm/amdgpu: validate bad page threshold in ras drm/amdgpu: conduct bad gpu check during bootup/reset drm/amdgpu: restore ras flags when user resets eeprom drm/amdgpu: calculate actual size instead of hardcode size
drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 +++- drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++ drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 70 ++++++++++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 19 +++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 98 ++++++++++++++++++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 8 +- 7 files changed, 211 insertions(+), 17 deletions(-)
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 1/5 drm/amdgpu: add bad page count threshold in module parameter
bad_page_threshold could be specified to detect and retire bad GPU if faulty bad pages exceed it.
When it's -1, ras will use typical bad page failure value.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -181,6 +181,7 @@ extern uint amdgpu_dm_abm_level;extern struct amdgpu_mgpu_info mgpu_info;extern int amdgpu_ras_enable;extern uint amdgpu_ras_mask; +extern int amdgpu_bad_page_threshold;extern int amdgpu_async_gfx_ring;extern int amdgpu_mcbp;extern int amdgpu_discovery;
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +11 -0
@@ -161,6 +161,7 @@ struct amdgpu_mgpu_info mgpu_info = { };int amdgpu_ras_enable =-1 ;uint amdgpu_ras_mask =0xffffffff ; +int amdgpu_bad_page_threshold =-1 ;/** * DOC: vramlimit (int)
@@ -801,6 +802,16 @@ module_param_named(tmz, amdgpu_tmz, int, 0444);MODULE_PARM_DESC (reset_method,"GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)" );module_param_named (reset_method, amdgpu_reset_method, int,0444 ); +/** +* DOC: bad_page_threshold (int) +* Bad page threshold configuration is driven by RMA(Return Merchandise +* Authorization) policy, which is to specify the threshold value of faulty +* pages detected by ECC, which may result in GPU's retirement if total +* faulty pages by ECC exceed threshold value. +*/ +MODULE_PARM_DESC (bad_page_threshold,"Bad page threshold(-1 = auto(default typical value))" ); +module_param_named (bad_page_threshold, amdgpu_bad_page_threshold, int,0444 ); +static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002, 0x6780, PCI_ANY_ID, PCI_ANY_ID, 0, 0, CHIP_TAHITI},
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
[AMD Official Use Only - Internal Distribution Only]
Hi, Guchun,
It is better to let user be able to change amdgpu_bad_page_threshold with sysfs, so that users no need to reboot system when they want to change their strategy.
Best Regards
Dennis Li
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
bad_page_threshold could be specified to detect and retire bad GPU if faulty bad pages exceed it.
When it's -1, ras will use typical bad page failure value.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -181,6 +181,7 @@ extern uint amdgpu_dm_abm_level; extern struct amdgpu_mgpu_info mgpu_info; extern int amdgpu_ras_enable; extern uint amdgpu_ras_mask; +extern int amdgpu_bad_page_threshold;extern int amdgpu_async_gfx_ring;extern int amdgpu_mcbp;extern int amdgpu_discovery;
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +13 -1
@@ -161,6 +161,7 @@ struct amdgpu_mgpu_info mgpu_info = { }; int amdgpu_ras_enable = -1; uint amdgpu_ras_mask = 0xffffffff; +int amdgpu_bad_page_threshold =-1 ;/** * DOC: vramlimit (int)
@@ -801,6 +802,16 @@ module_param_named(tmz, amdgpu_tmz, int, 0444); MODULE_PARM_DESC(reset_method, "GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)"); module_param_named(reset_method, amdgpu_reset_method, int, 0444); +/** +* DOC: bad_page_threshold (int) +* Bad page threshold configuration is driven by RMA(Return Merchandise +* Authorization) policy, which is to specify the threshold value of +faulty +* pages detected by ECC, which may result in GPU's retirement if total +* faulty pages by ECC exceed threshold value. +*/ +MODULE_PARM_DESC(bad_page_threshold, "Bad page threshold(-1 = +auto(default typical value))"); module_param_named(bad_page_threshold, +amdgpu_bad_page_threshold, int, 0444); + static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002 ,0x6780 ,PCI_ANY_ID ,PCI_ANY_ID ,0 ,0 ,CHIP_TAHITI }, --
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
[AMD Public Use]
Hi Dennis,
To be honest, your suggestion is considered when I start the design. My thought is in actual world, bad page threshold is one static configuration, it should be set once when probing. So module parameter is one ideal choice for this.
Regards,
Guchun
-----Original Message-----
From: Li, Dennis <[email protected]>
Sent: Thursday, July 23, 2020 8:32 AM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
[AMD Official Use Only - Internal Distribution Only]
Hi, Guchun,
It is better to let user be able to change amdgpu_bad_page_threshold with sysfs, so that users no need to reboot system when they want to change their strategy.
Best Regards
Dennis Li
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
bad_page_threshold could be specified to detect and retire bad GPU if faulty bad pages exceed it.
When it's -1, ras will use typical bad page failure value.
--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++++++++++ 2 files changed, 12 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu.h +1 -0
@@ -181,6 +181,7 @@ extern uint amdgpu_dm_abm_level; extern struct amdgpu_mgpu_info mgpu_info; extern int amdgpu_ras_enable; extern uint amdgpu_ras_mask; +extern int amdgpu_bad_page_threshold;extern int amdgpu_async_gfx_ring;extern int amdgpu_mcbp;extern int amdgpu_discovery;
drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +13 -1
@@ -161,6 +161,7 @@ struct amdgpu_mgpu_info mgpu_info = { }; int amdgpu_ras_enable = -1; uint amdgpu_ras_mask = 0xffffffff; +int amdgpu_bad_page_threshold =-1 ;/** * DOC: vramlimit (int)
@@ -801,6 +802,16 @@ module_param_named(tmz, amdgpu_tmz, int, 0444); MODULE_PARM_DESC(reset_method, "GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)"); module_param_named(reset_method, amdgpu_reset_method, int, 0444); +/** +* DOC: bad_page_threshold (int) +* Bad page threshold configuration is driven by RMA(Return Merchandise +* Authorization) policy, which is to specify the threshold value of +faulty +* pages detected by ECC, which may result in GPU's retirement if total +* faulty pages by ECC exceed threshold value. +*/ +MODULE_PARM_DESC(bad_page_threshold, "Bad page threshold(-1 = +auto(default typical value))"); module_param_named(bad_page_threshold, +amdgpu_bad_page_threshold, int, 0444); + static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002 ,0x6780 ,PCI_ANY_ID ,PCI_ANY_ID ,0 ,0 ,CHIP_TAHITI }, --
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
I agree with Guchun as well.
When you have a dynamic module parameter and change the bad page threshold the GPU might just stop working suddenly.
That is not a good idea as far as I can see.
Regards,
Christian.
Am 23.07.20 um 05:47 schrieb Chen, Guchun:
Quoted 75 lines
[AMD Public Use]
Hi Dennis,
To be honest, your suggestion is considered when I start the design. My thought is in actual world, bad page threshold is one static configuration, it should be set once when probing. So module parameter is one ideal choice for this.
Regards,
Guchun-----Original Message-----
From: Li, Dennis <[email protected]>
Sent: Thursday, July 23, 2020 8:32 AM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter[AMD Official Use Only - Internal Distribution Only]
Hi, Guchun,
It is better to let user be able to changeamdgpu_bad_page_thresholdwith sysfs, so that users no need to reboot system when they want to change their strategy.Best Regards
Dennis Li
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
bad_page_thresholdcould be specified to detect and retire bad GPU if faulty bad pages exceed it.When it's -1, ras will use typical bad page failure value.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++++++++++ 2 files changed, 12 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -181,6 +181,7 @@ extern uint amdgpu_dm_abm_level; extern struct amdgpu_mgpu_info mgpu_info; extern int amdgpu_ras_enable; extern uint amdgpu_ras_mask; +extern int amdgpu_bad_page_threshold;extern int amdgpu_async_gfx_ring;extern int amdgpu_mcbp;extern int amdgpu_discovery;drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +13
-1@@ -161,6 +161,7 @@ struct amdgpu_mgpu_info mgpu_info = { }; int amdgpu_ras_enable = -1; uint amdgpu_ras_mask = 0xffffffff; +int amdgpu_bad_page_threshold =-1 ;/** * DOC: vramlimit (int) @@ -801,6 +802,16 @@ module_param_named(tmz, amdgpu_tmz, int, 0444); MODULE_PARM_DESC(reset_method, "GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)"); module_param_named(reset_method, amdgpu_reset_method, int, 0444); +/** +* DOC: bad_page_threshold (int) +* Bad page threshold configuration is driven by RMA(Return Merchandise +* Authorization) policy, which is to specify the threshold value of +faulty +* pages detected by ECC, which may result in GPU's retirement if total +* faulty pages by ECC exceed threshold value. +*/ +MODULE_PARM_DESC(bad_page_threshold, "Bad page threshold(-1 = +auto(default typical value))"); module_param_named(bad_page_threshold, +amdgpu_bad_page_threshold, int, 0444); + static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002 ,0x6780 ,PCI_ANY_ID ,PCI_ANY_ID ,0 ,0 ,CHIP_TAHITI }, --
Quoted 5 lines
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
Also note that module parameters are global. If you change the parameter, it changes it for all GPUs in the system. That may not be what the customer wants.
Alex
Quoted 89 lines · Christian König
I agree with Guchun as well.
When you have a dynamic module parameter and change the bad page threshold the GPU might just stop working suddenly.
That is not a good idea as far as I can see.
Regards,
Christian.Am 23.07.20 um 05:47 schrieb Chen, Guchun:
Quoted 77 lines
[AMD Public Use]
Hi Dennis,
To be honest, your suggestion is considered when I start the design. My thought is in actual world, bad page threshold is one static configuration, it should be set once when probing. So module parameter is one ideal choice for this.
Regards,
Guchun-----Original Message-----
From: Li, Dennis <[email protected]>
Sent: Thursday, July 23, 2020 8:32 AM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter[AMD Official Use Only - Internal Distribution Only]
Hi, Guchun,
It is better to let user be able to changeamdgpu_bad_page_thresholdwith sysfs, so that users no need to reboot system when they want to change their strategy.Best Regards
Dennis Li
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 1/5] drm/amdgpu: add bad page count threshold in module parameter
bad_page_thresholdcould be specified to detect and retire bad GPU if faulty bad pages exceed it.When it's -1, ras will use typical bad page failure value.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu.h | 1 + drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c | 11 +++++++++++ 2 files changed, 12 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu.h +1
-0@@ -181,6 +181,7 @@ extern uint amdgpu_dm_abm_level; extern struct amdgpu_mgpu_info mgpu_info; extern int amdgpu_ras_enable; extern uint amdgpu_ras_mask; +extern int amdgpu_bad_page_threshold;extern int amdgpu_async_gfx_ring;extern int amdgpu_mcbp;extern int amdgpu_discovery;drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c +13
-1@@ -161,6 +161,7 @@ struct amdgpu_mgpu_info mgpu_info = { }; int amdgpu_ras_enable = -1; uint amdgpu_ras_mask = 0xffffffff; +int amdgpu_bad_page_threshold =-1 ;/** * DOC: vramlimit (int) @@ -801,6 +802,16 @@ module_param_named(tmz, amdgpu_tmz, int, 0444); MODULE_PARM_DESC(reset_method, "GPU reset method (-1 = auto (default), 0 = legacy, 1 = mode0, 2 = mode1, 3 = mode2, 4 = baco)"); module_param_named(reset_method, amdgpu_reset_method, int, 0444); +/** +* DOC: bad_page_threshold (int) +* Bad page threshold configuration is driven by RMA(Return Merchandise +* Authorization) policy, which is to specify the threshold value of +faulty +* pages detected by ECC, which may result in GPU's retirement if total +* faulty pages by ECC exceed threshold value. +*/ +MODULE_PARM_DESC(bad_page_threshold, "Bad page threshold(-1 = +auto(default typical value))"); module_param_named(bad_page_threshold, +amdgpu_bad_page_threshold, int, 0444); + static const struct pci_device_id pciidlist[] = { #ifdef CONFIG_DRM_AMDGPU_SI {0x1002 ,0x6780 ,PCI_ANY_ID ,PCI_ANY_ID ,0 ,0 ,CHIP_TAHITI }, --2.17.1
_______________________________________________
Quoted 8 lines
Quoted 3 lines
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 2/5 drm/amdgpu: validate bad page threshold in ras
In reply to Guchun ChenBad page threshold value should be valid in the range between -1 and max records length of eeprom. It could determine when the GPU should be retired.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 43 +++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 3 ++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 5 +++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 2 + 4 files changed, 53 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +43 -0
@@ -69,6 +69,9 @@ const char *ras_block_string[] = {/* inject address is 52 bits */ #define RAS_UMC_INJECT_ADDR_LIMIT (0x1ULL << 52) +/* typical ECC bad page rate(1 bad page per 100MB VRAM) */ +#define RAS_BAD_PAGE_RATE (100 * 1024 * 1024ULL) +enum amdgpu_ras_retire_page_reservation {AMDGPU_RAS_RETIRE_PAGE_RESERVED ,AMDGPU_RAS_RETIRE_PAGE_PENDING ,
@@ -1700,6 +1703,42 @@ static bool amdgpu_ras_check_bad_page(struct amdgpu_device *adev,return ret; } +static void amdgpu_ras_validate_threshold (struct amdgpu_device *adev, +uint32_t max_length) +{ +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +int tmp_threshold = amdgpu_bad_page_threshold; +u64 val; + +/* +* Justification of value bad_page_cnt_threshold in ras structure +* +* 1. -1 <= amdgpu_bad_page_threshold <= max record length in eeprom +* 2. if amdgpu_bad_page_threshold = -1, +* bad_page_cnt_threshold = typical value by formula. +* 3. if amdgpu_bad_page_threshold = 0, +* bad_page_cnt_threshold = 0xFFFFFFFF, +* and disable RMA feature accordingly. +* 4. use the value specified from user when (amdgpu_bad_page_threshold +* > 0 && < max record length in eeprom). +*/ + +if (tmp_threshold <-1 ) + tmp_threshold =-1 ; +else if (tmp_threshold > max_length) + tmp_threshold = max_length; + +if (tmp_threshold ==-1 ) { + val = adev->gmc.mc_vram_size; +do_div (val,RAS_BAD_PAGE_RATE ); + con->bad_page_cnt_threshold =lower_32_bits (val); + }else if (tmp_threshold ==0 ) { + con->bad_page_cnt_threshold =0xFFFFFFFF ; + }else { + con->bad_page_cnt_threshold = tmp_threshold; + } +} +/* called in gpu recovery/init */ int amdgpu_ras_reserve_bad_pages (struct amdgpu_device *adev) {
@@ -1777,6 +1816,7 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) {struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data; +uint32_t max_eeprom_records_len =0 ;int ret;if (con)
@@ -1795,6 +1835,9 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev)atomic_set (&con->in_recovery,0 ); con->adev = adev; + max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length (); +amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); + ret =amdgpu_ras_eeprom_init (&con->eeprom_control);if (ret) goto free;
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +3 -0
@@ -336,6 +336,9 @@ struct amdgpu_ras {struct amdgpu_ras_eeprom_control eeprom_control;bool error_query_ready; + +/* bad page count threshold */ +uint32_t bad_page_cnt_threshold; };struct ras_fs_data {
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5 -0
@@ -499,6 +499,11 @@ int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control *control,return ret == num ?0 : -EIO ; } +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ) +{ +return EEPROM_MAX_RECORD_NUM ; +} +/* Used for testing if bugs encountered */ #if 0 void amdgpu_ras_eeprom_test (struct amdgpu_ras_eeprom_control *control)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +2 -0
@@ -85,6 +85,8 @@ int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control *control,bool write, int num); +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ); +void amdgpu_ras_eeprom_test (struct amdgpu_ras_eeprom_control *control);#endif // _AMDGPU_RAS_EEPROM_H
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 2/5] drm/amdgpu: validate bad page threshold in ras
[AMD Official Use Only - Internal Distribution Only]
Hi Guchun,
Please see my comment inline.
Quoted 65 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 2/5] drm/amdgpu: validate bad page threshold in rasBad page threshold value should be valid in the range between -1 and max records length of eeprom. It could determine when the GPU should be retired.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 43 +++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 3 ++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 5 +++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 2 + 4 files changed, 53 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +14
-0@@ -69,6 +69,9 @@ const char *ras_block_string[] = {/* inject address is 52 bits */ #define RAS_UMC_INJECT_ADDR_LIMIT (0x1ULL << 52) +/* typical ECC bad page rate(1 bad page per 100MB VRAM) */ +#define RAS_BAD_PAGE_RATE (100 * 1024 * 1024ULL) +enum amdgpu_ras_retire_page_reservation {AMDGPU_RAS_RETIRE_PAGE_RESERVED ,AMDGPU_RAS_RETIRE_PAGE_PENDING ,@@ -1700,6 +1703,42 @@ static bool amdgpu_ras_check_bad_page(structamdgpu_device *adev, return ret; } +static void amdgpu_ras_validate_threshold (struct amdgpu_device *adev, +uint32_t max_length) +{ +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +int tmp_threshold = amdgpu_bad_page_threshold; +u64 val; + +/* +* Justification of value bad_page_cnt_threshold in ras structure +* +* 1. -1 <= amdgpu_bad_page_threshold <= max record length in eeprom
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +6
-0+* 2. if amdgpu_bad_page_threshold = -1, +* bad_page_cnt_threshold = typical value by formula. +* 3. if amdgpu_bad_page_threshold = 0, +* bad_page_cnt_threshold = 0xFFFFFFFF, +* and disable RMA feature accordingly. +* 4. use the value specified from user when (
amdgpu_bad_page_thresholddrivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +2
-0+* > 0 && < max record length in eeprom). +*/
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c static bool amdgpu_ras_check_bad_page(struct amdgpu_device *adev, · line 1734
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +10
-0+ +if (tmp_threshold <-1 ) + tmp_threshold =-1 ; +else if (tmp_threshold > max_length) + tmp_threshold = max_length; + +if (tmp_threshold ==-1 ) { + val = adev->gmc.mc_vram_size; +do_div (val,RAS_BAD_PAGE_RATE ); + con->bad_page_cnt_threshold =lower_32_bits (val);
[Yang, Stanley] : It's better to compare con->bad_page_cnt_threshold with max_length, the value of bad_page_cnt_threshold should not exceed max_length.
Quoted 76 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +7
-0+ }else if (tmp_threshold ==0 ) { + con->bad_page_cnt_threshold =0xFFFFFFFF ; + }else { + con->bad_page_cnt_threshold = tmp_threshold; + } +} +/* called in gpu recovery/init */ int amdgpu_ras_reserve_bad_pages(struct amdgpu_device *adev) { @@ -1777,6 +1816,7 @@ int
amdgpu_ras_recovery_init(struct amdgpu_device *adev){drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +4
-0struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data; +uint32_t max_eeprom_records_len =0 ;int ret;if (con)@@ -1795,6 +1835,9 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) atomic_set(&con->in_recovery, 0); con->adev = adev; + max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length (); +amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); + ret =amdgpu_ras_eeprom_init (&con->eeprom_control);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +0
-0if (ret) goto free;drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +3
-0@@ -336,6 +336,9 @@ struct amdgpu_ras {struct amdgpu_ras_eeprom_control eeprom_control;bool error_query_ready; + +/* bad page count threshold */ +uint32_t bad_page_cnt_threshold; };struct ras_fs_data {drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5
-0@@ -499,6 +499,11 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, return ret == num ? 0 : -EIO; } +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ) +{ +return EEPROM_MAX_RECORD_NUM ; +} + /* Used for testing if bugs encountered */ #if 0 void amdgpu_ras_eeprom_test(struct amdgpu_ras_eeprom_control *control)drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +2
-0@@ -85,6 +85,8 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, bool write, int num); +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ); +void amdgpu_ras_eeprom_test (struct amdgpu_ras_eeprom_control *control);
Quoted 3 lines
#endif //
_AMDGPU_RAS_EEPROM_H
--
2.17.1
Regards,
Stanley
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 2/5] drm/amdgpu: validate bad page threshold in ras
[AMD Public Use]
Thanks for review, Stanley.
Re: [Yang, Stanley] : It's better to compare con->bad_page_cnt_threshold with max_length, the value of bad_page_cnt_threshold should not exceed max_length.
Correct, one guard is necessary. It will be patch v2.
Regards,
Guchun
-----Original Message-----
From: Yang, Stanley <[email protected]>
Sent: Wednesday, July 22, 2020 3:52 PM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 2/5] drm/amdgpu: validate bad page threshold in ras
[AMD Official Use Only - Internal Distribution Only]
Hi Guchun,
Please see my comment inline.
Quoted 64 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 2/5] drm/amdgpu: validate bad page threshold in rasBad page threshold value should be valid in the range between -1 and max records length of eeprom. It could determine when the GPU should be retired.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 43 +++++++++++++++++++ drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 3 ++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 5 +++ .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 2 + 4 files changed, 53 insertions(+)drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +14
-0@@ -69,6 +69,9 @@ const char *ras_block_string[] = {/* inject address is 52 bits */ #define RAS_UMC_INJECT_ADDR_LIMIT (0x1ULL << 52) +/* typical ECC bad page rate(1 bad page per 100MB VRAM) */ +#define RAS_BAD_PAGE_RATE (100 * 1024 * 1024ULL) +enum amdgpu_ras_retire_page_reservation {AMDGPU_RAS_RETIRE_PAGE_RESERVED ,AMDGPU_RAS_RETIRE_PAGE_PENDING ,@@ -1700,6 +1703,42 @@ static bool amdgpu_ras_check_bad_page(structamdgpu_device *adev, return ret; } +static void amdgpu_ras_validate_threshold (struct amdgpu_device *adev, +uint32_t max_length) +{ +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +int tmp_threshold = amdgpu_bad_page_threshold; +u64 val; + +/* +* Justification of value bad_page_cnt_threshold in ras structure +* +* 1. -1 <= amdgpu_bad_page_threshold <= max record length in eeprom
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +6
-0+* 2. if amdgpu_bad_page_threshold = -1, +* bad_page_cnt_threshold = typical value by formula. +* 3. if amdgpu_bad_page_threshold = 0, +* bad_page_cnt_threshold = 0xFFFFFFFF, +* and disable RMA feature accordingly. +* 4. use the value specified from user when (
amdgpu_bad_page_thresholddrivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +2
-0+* > 0 && < max record length in eeprom). +*/
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c static bool amdgpu_ras_check_bad_page(struct amdgpu_device *adev, · line 1734
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +10
-0+ +if (tmp_threshold <-1 ) + tmp_threshold =-1 ; +else if (tmp_threshold > max_length) + tmp_threshold = max_length; + +if (tmp_threshold ==-1 ) { + val = adev->gmc.mc_vram_size; +do_div (val,RAS_BAD_PAGE_RATE ); + con->bad_page_cnt_threshold =lower_32_bits (val);
[Yang, Stanley] : It's better to compare con->bad_page_cnt_threshold with max_length, the value of bad_page_cnt_threshold should not exceed max_length.
Quoted 76 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +7
-0+ }else if (tmp_threshold ==0 ) { + con->bad_page_cnt_threshold =0xFFFFFFFF ; + }else { + con->bad_page_cnt_threshold = tmp_threshold; + } +} +/* called in gpu recovery/init */ int amdgpu_ras_reserve_bad_pages(struct amdgpu_device *adev) { @@ -1777,6 +1816,7 @@ int
amdgpu_ras_recovery_init(struct amdgpu_device *adev){drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +4
-0struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data; +uint32_t max_eeprom_records_len =0 ;int ret;if (con)@@ -1795,6 +1835,9 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) atomic_set(&con->in_recovery, 0); con->adev = adev; + max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length (); +amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); + ret =amdgpu_ras_eeprom_init (&con->eeprom_control);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +0
-0if (ret) goto free;drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +3
-0@@ -336,6 +336,9 @@ struct amdgpu_ras {struct amdgpu_ras_eeprom_control eeprom_control;bool error_query_ready; + +/* bad page count threshold */ +uint32_t bad_page_cnt_threshold; };struct ras_fs_data {drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5
-0@@ -499,6 +499,11 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, return ret == num ? 0 : -EIO; } +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ) +{ +return EEPROM_MAX_RECORD_NUM ; +} + /* Used for testing if bugs encountered */ #if 0 void
amdgpu_ras_eeprom_test(struct amdgpu_ras_eeprom_control *control)diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.hb/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h
index7e8647a05df7..b272840cb069100644drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +2
-0@@ -85,6 +85,8 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, bool write, int num); +inline uint32_t amdgpu_ras_eeprom_get_record_max_length (void ); +void amdgpu_ras_eeprom_test (struct amdgpu_ras_eeprom_control *control);
Quoted 3 lines
#endif //
_AMDGPU_RAS_EEPROM_H
--
2.17.1
Regards,
Stanley
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 3/5 drm/amdgpu: conduct bad gpu check during bootup/reset
In reply to Guchun ChenDuring boot up, when init eeprom, RAS will check if the BAD GPU tag is saved or not. And will break boot up and notify user to RMA it.
At the meanwhile, when saved bad page count to eeprom exceeds threshold, one ras recovery will be triggered immediately, and bad gpu check will be executed and reset will fail as well to remind user.
User could set bad_page_threshold = 0 as module parameter when probing driver to disable the bad feature check.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 25 +++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 16 +++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 87 ++++++++++++++++++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 6 +- 5 files changed, 142 insertions(+), 13 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +17 -4
@@ -2065,7 +2065,9 @@ static int amdgpu_device_ip_init(struct amdgpu_device *adev)* Note: theoretically, this should be called before all vram allocations * to protect retired page from abusing */ -amdgpu_ras_recovery_init (adev); + r =amdgpu_ras_recovery_init (adev); +if (r) + goto init_failed;if (adev->gmc.xgmi.num_physical_nodes >1 )amdgpu_xgmi_add_device (adev);
@@ -4133,8 +4135,20 @@ static int amdgpu_do_asic_reset(struct amdgpu_hive_info *hive,amdgpu_fbdev_set_suspend (tmp_adev,0 ); -/* must succeed. */ -amdgpu_ras_resume (tmp_adev); +/* +* The CPU is BAD once faulty pages by ECC has +* reached the threshold, and ras recovery is +* scheduled. So add one check here to prevent +* recovery if it's one BAD GPU, and remind +* user to RMA this GPU. +*/ +if (!amdgpu_ras_check_bad_gpu (tmp_adev)) { +/* must succeed. */ +amdgpu_ras_resume (tmp_adev); + }else { + r = -EINVAL ; + goto out; + }/* Update PSP FW topology after reset */ if (hive && tmp_adev->gmc.xgmi.num_physical_nodes >1 )
@@ -4142,7 +4156,6 @@ static int amdgpu_do_asic_reset(struct amdgpu_hive_info *hive, } } - out:if (!r) {amdgpu_irq_gpu_reset_resume_helper (tmp_adev);
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +21 -4
@@ -62,8 +62,6 @@ const char *ras_block_string[] = {#define ras_err_str (i) (ras_error_string[ffs(i)])#define ras_block_str (i) (ras_block_string[i]) -#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS 1 -#define AMDGPU_RAS_FLAG_INIT_NEED_RESET 2#define RAS_DEFAULT_FLAGS (AMDGPU_RAS_FLAG_INIT_BY_VBIOS)/* inject address is 52 bits */
@@ -1817,6 +1815,7 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev)struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data;uint32_t max_eeprom_records_len =0 ; +bool bad_gpu = false;int ret;if (con)
@@ -1838,9 +1837,14 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length ();amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); - ret =amdgpu_ras_eeprom_init (&con->eeprom_control); -if (ret) + ret =amdgpu_ras_eeprom_init (&con->eeprom_control, &bad_gpu); +/* +* we only fail this calling and halt booting when bad_gpu is true. +*/ +if (bad_gpu) { + ret = -EINVAL ; goto free; + }if (con->eeprom_control.num_recs) { ret =amdgpu_ras_load_bad_pages (adev);
@@ -2189,3 +2193,16 @@ bool amdgpu_ras_need_emergency_restart(struct amdgpu_device *adev)return false; } + +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev) +{ +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +bool bad_gpu = false; + +if (con && (con->bad_page_cnt_threshold !=0xFFFFFFFF )) +amdgpu_ras_eeprom_check_bad_gpu (&con->eeprom_control, + &bad_gpu); + +/* We are only interested in variable bad_gpu. */ +return bad_gpu; +}
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +13 -3
@@ -31,6 +31,10 @@#include "ta_ras_if.h" #include "amdgpu_ras_eeprom.h" +#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS (0x1 << 0) +#define AMDGPU_RAS_FLAG_INIT_NEED_RESET (0x1 << 1) +#define AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV (0x1 << 2) +enum amdgpu_ras_block {AMDGPU_RAS_BLOCK__UMC =0 ,AMDGPU_RAS_BLOCK__SDMA ,
@@ -493,6 +497,8 @@ void amdgpu_ras_suspend(struct amdgpu_device *adev);unsigned long amdgpu_ras_query_error_count (struct amdgpu_device *adev,bool is_ce); +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev); +/* error handling functions */ int amdgpu_ras_add_bad_pages (struct amdgpu_device *adev,struct eeprom_table_record *bps,int pages);
@@ -503,10 +509,14 @@ static inline int amdgpu_ras_reset_gpu(struct amdgpu_device *adev) {struct amdgpu_ras *ras =amdgpu_ras_get_context (adev); -/* save bad page to eeprom before gpu reset, -* i2c may be unstable in gpu reset +/* +* Save bad page to eeprom before gpu reset, i2c may be unstable +* in gpu reset. +* +* Also, exclude the case when ras recovery issuer is +* eerprom page write. */ -if (in_task ()) +if (!(ras->flags &AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV ) &&in_task ())amdgpu_ras_reserve_bad_pages (adev);if (atomic_cmpxchg (&ras->in_recovery,0 ,1 ) ==0 )
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +86 -1
142 lines not on this page · raw message
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +5 -1
@@ -77,9 +77,13 @@ struct eeprom_table_record {unsigned char mcumc_id; }__attribute__((__packed__)); -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu);int amdgpu_ras_eeprom_reset_table (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_check_bad_gpu (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu); + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control *control, struct eeprom_table_record *records, bool write,
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset
[AMD Official Use Only - Internal Distribution Only]
Quoted 36 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/resetDuring boot up, when init eeprom, RAS will check if the BAD GPU tag is saved or not. And will break boot up and notify user to RMA it.
At the meanwhile, when saved bad page count to eeprom exceeds threshold, one ras recovery will be triggered immediately, and bad gpu check will be executed and reset will fail as well to remind user.
User could set
bad_page_threshold= 0 as module parameter when probing driver to disable the bad feature check.Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 25 +++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 16 +++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 87 ++++++++++++++++++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 6 +- 5 files changed, 142 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2068
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -2065,7 +2065,9 @@ static int amdgpu_device_ip_init(structamdgpu_device *adev) * Note: theoretically, this should be called before all vram allocations * to protect retired page from abusing */
[Tao] The comment above should be also updated
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct amdgpu_device *adev) · line 2070
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3
-1-amdgpu_ras_recovery_init (adev); +r =amdgpu_ras_recovery_init (adev); +if (r) +goto init_failed;
[Tao] Are you sure "r != 0" equals to "bad gpu"? Other errors of recovery_init should not block gpu init.
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c
Quoted 144 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14
-2if (adev->gmc.xgmi.num_physical_nodes >1 )amdgpu_xgmi_add_device (adev);@@ -4133,8 +4135,20 @@ static int amdgpu_do_asic_reset(struct amdgpu_hive_info *hive,amdgpu_fbdev_set_suspend (tmp_adev,0 ); -/* must succeed. */ -amdgpu_ras_resume (tmp_adev); +/* +* The CPU is BAD once faulty pages by ECC has +* reached the threshold, and ras recovery is +* scheduled. So add one check here to prevent +* recovery if it's one BAD GPU, and remind +* user to RMA this GPU. +*/ +if (!amdgpu_ras_check_bad_gpu (tmp_adev)) { +/* must succeed. */ +amdgpu_ras_resume (tmp_adev); +}else { +r = -EINVAL ; +goto out; +}/* Update PSP FW topology after reset */ if (hive && tmp_adev-
Quoted 1 line
gmc.xgmi.num_physical_nodes > 1) @@ -4142,7 +4156,6 @@ static int
amdgpu_do_asic_reset(struct
amdgpu_hive_info*hive,
}
}drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-1- out:if (!r) {amdgpu_irq_gpu_reset_resume_helper (tmp_adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +1
-2@@ -62,8 +62,6 @@ const char *ras_block_string[] = { #define ras_err_str(i) (ras_error_string[ffs(i)]) #define ras_block_str(i) (ras_block_string[i]) -#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS1 -#define AMDGPU_RAS_FLAG_INIT_NEED_RESET2 #define RAS_DEFAULT_FLAGS (AMDGPU_RAS_FLAG_INIT_BY_VBIOS)/* inject address is 52 bits */ @@ -1817,6 +1815,7 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev)struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data;uint32_t max_eeprom_records_len =0 ; +bool bad_gpu = false;int ret;if (con)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +7
-2@@ -1838,9 +1837,14 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length ();amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); -ret =amdgpu_ras_eeprom_init (&con->eeprom_control); -if (ret) +ret =amdgpu_ras_eeprom_init (&con->eeprom_control, &bad_gpu); +/* +* we only fail this calling and halt booting when bad_gpu is true. +*/ +if (bad_gpu) { +ret = -EINVAL ; goto free; +}if (
con->eeprom_control.num_recs) {
ret =amdgpu_ras_load_bad_pages(adev); @@ -2189,3
+2193,16 @@ boolamdgpu_ras_need_emergency_restart(struct amdgpu_device *adev)return false;
}drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +12
-0+ +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev) { +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +bool bad_gpu = false; + +if (con && (con->bad_page_cnt_threshold !=0xFFFFFFFF )) +amdgpu_ras_eeprom_check_bad_gpu (&con->eeprom_control, +&bad_gpu); + +/* We are only interested in variable bad_gpu. */ +return bad_gpu; +}drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +6
-0@@ -31,6 +31,10 @@#include "ta_ras_if.h" #include "amdgpu_ras_eeprom.h" +#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS(0x1 << 0) +#define AMDGPU_RAS_FLAG_INIT_NEED_RESET(0x1 << 1) +#define AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV(0x1 << 2) +enum amdgpu_ras_block {AMDGPU_RAS_BLOCK__UMC =0 ,AMDGPU_RAS_BLOCK__SDMA ,@@ -493,6 +497,8 @@ void amdgpu_ras_suspend(struct amdgpu_device *adev); unsigned long amdgpu_ras_query_error_count(struct amdgpu_device *adev,bool is_ce); +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev); +/* error handling functions */ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,struct
eeprom_table_record*bps, int pages); @@ -503,10
+509,14 @@ static inline intamdgpu_ras_reset_gpu(struct amdgpu_device *adev){
structamdgpu_ras*ras =amdgpu_ras_get_context(adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +7
-3-/* save bad page to eeprom before gpu reset, -* i2c may be unstable in gpu reset +/* +* Save bad page to eeprom before gpu reset, i2c may be unstable +* in gpu reset. +* +* Also, exclude the case when ras recovery issuer is +* eerprom page write. */ -if (in_task ()) +if (!(ras->flags & AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV) &&
in_task())
amdgpu_ras_reserve_bad_pages(adev);if (
atomic_cmpxchg(&ras->in_recovery, 0, 1)== 0) diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cindexa2c982b1eac6..96b63c026bad100644drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0
[Tao] It's better to split it into two patches, one is for ras and another one is for eeprom
Quoted 72 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0@@ -46,6 +46,9 @@#define EEPROM_TABLE_HDR_VAL 0x414d4452 #define EEPROM_TABLE_VER
0x00010000drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-1+/* Bad GPU tag ‘BADG’ */ +#define EEPROM_TABLE_HDR_BAD 0x42414447 +/* Assume 2 Mbit size */ #define EEPROM_SIZE_BYTES 256000#define EEPROM_PAGE__SIZE_BYTES 256@@ -238,12 +241,14 @@ int amdgpu_ras_eeprom_reset_table(struct amdgpu_ras_eeprom_control *control) } -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control) +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu) {int ret =0 ;struct amdgpu_device *adev =to_amdgpu_device (control);unsigned char buff[EEPROM_ADDRESS_SIZE +EEPROM_TABLE_HEADER_SIZE ] = {0 };struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev);struct i2c_msg msg = {.addr= 0,
.flags=I2C_M_RD,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -251,6 +256,8 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) .buf= buff, }; +*bad_gpu = false; +/* Verify i2c adapter is initialized */ if (!adev->pm.smu_i2c.algo)return
-ENOENT;drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5
-0@@ -279,6 +286,11 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) DRM_DEBUG_DRIVER("Found existing EEPROM table with %d records", control->num_recs); +}else if ((ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && ( +hdr->header ==EEPROM_TABLE_HDR_BAD )) { +*bad_gpu = true; +DRM_ERROR ("Detect BAD GPU TAG in eeprom table and " +"GPU shall be RMAed.\n" ); }else {
DRM_INFO("Creating new EEPROM table");drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +13
-0@@ -375,6 +387,44 @@ static uint32_t __correct_eeprom_dest_address(uint32_t curr_address) return curr_address; } +int amdgpu_ras_eeprom_check_bad_gpu (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu) +{ +struct amdgpu_device *adev =to_amdgpu_device (control); +unsigned char buff[EEPROM_ADDRESS_SIZE + +EEPROM_TABLE_HEADER_SIZE ] = {0 }; +struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; +struct i2c_msg msg = { +.addr = control->i2c_address, +.flags =I2C_M_RD , +.len =EEPROM_ADDRESS_SIZE +EEPROM_TABLE_HEADER_SIZE , +.buf = buff, +};
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c static uint32_t · line 415
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +10
-0+int ret; + +*bad_gpu = false; + +/* read EEPROM table header */ +mutex_lock (&control->tbl_mutex); +ret =i2c_transfer (&adev->pm.smu_i2c, &msg,1 ); +if (ret <1 ) { +dev_err (adev->dev,"Failed to read EEPROM table header.\n" ); +goto err;
[Tao] One tab is missed
Quoted 108 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-0+} + +__decode_table_header_from_buff (hdr, &buff[2 ]); + +if (hdr->header ==EEPROM_TABLE_HDR_BAD ) { +dev_warn(adev->dev, "Current GPU is BAD and should beRMAed.\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +9
-0+*bad_gpu = true; +} + +ret =0 ; +err: +mutex_unlock (&control->tbl_mutex); +return ret; +} + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control*control,
structeeprom_table_record
*records,
bool write,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -383,8 +433,10 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, int i, ret = 0;struct i2c_msg *msgs, *msg;unsigned char *buffs, *buff; +bool sched_ras_recovery = false;struct eeprom_table_record *record;struct amdgpu_device *adev =to_amdgpu_device (control); +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev); if (adev->asic_type != CHIP_VEGA20 && adev->asic_type !=
CHIP_ARCTURUS)
return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +19
-0@@ -402,6 +454,25 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, goto free_buff; } +/* +* If saved bad pages number exceeds the bad page threshod for +* the whole VRAM, update table header to mark one BAD GPU and +* schedule one ras recovery after eeprom write is done, this +* can avoid the missing for latest records. +* +* This new header will be picked up and checked in the bootup by +* ras recovery, which may break bootup process to notify user this +* GPU is bad and to RMA(Return Merchandise Authorization) this GPU. +*/ +if (write && (ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && +((control->num_recs + num) >= ras->bad_page_cnt_threshold)) { +dev_warn (adev->dev, +"Saved bad pages(%d) reaches threshold value(%d).\n" , +control->num_recs + num, ras- >bad_page_cnt_threshold); +control->tbl_hdr.header =EEPROM_TABLE_HDR_BAD ; +sched_ras_recovery = true; +} +/* In case of overflow just start from beginning to not lose newest records */
if (write && (control->next_addr+EEPROM_TABLE_RECORD_SIZE*
num >EEPROM_SIZE_BYTES))
control->next_addr=EEPROM_RECORD_START; @@ -482,6
+553,20 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control*control,
__update_tbl_checksum(control, records, num,old_hdr_byte_sum);
__update_table_header(control, buffs);drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +11
-0+ +if (sched_ras_recovery) { +/* +* Before scheduling ras recovery, assert the related +* flag first, which shall bypass common bad page +* reservation execution in amdgpu_ras_reset_gpu. +*/ +amdgpu_ras_get_context (adev)->flags |= +AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV ; + +dev_warn(adev->dev, "Conduct ras recovery due to bad"
+"page threshold reached.\n");drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0+amdgpu_ras_reset_gpu (adev); +} }else if (!__validate_tbl_checksum (control, records, num)) {DRM_WARN ("EEPROM Table checksum mismatch!" );/* TODO Uncomment when EEPROM read/write is relliable */ drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +5
-1@@ -77,9 +77,13 @@ struct eeprom_table_record {unsigned char mcumc_id; }__attribute__((__packed__)); -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu);int amdgpu_ras_eeprom_reset_table (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_check_bad_gpu (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu); + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control
Quoted 6 lines
*control,
structeeprom_table_record
*records,
bool write,
--
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset
[AMD Public Use]
Thanks for your review, Tao.
Please check my comments after yours.
Regards,
Guchun
-----Original Message-----
From: Zhou1, Tao <[email protected]>
Sent: Thursday, July 23, 2020 10:51 AM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset
[AMD Official Use Only - Internal Distribution Only]
Quoted 36 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/resetDuring boot up, when init eeprom, RAS will check if the BAD GPU tag is saved or not. And will break boot up and notify user to RMA it.
At the meanwhile, when saved bad page count to eeprom exceeds threshold, one ras recovery will be triggered immediately, and bad gpu check will be executed and reset will fail as well to remind user.
User could set
bad_page_threshold= 0 as module parameter when probing driver to disable the bad feature check.Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 25 +++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 16 +++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 87 ++++++++++++++++++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 6 +- 5 files changed, 142 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2068
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -2065,7 +2065,9 @@ static int amdgpu_device_ip_init(structamdgpu_device *adev) * Note: theoretically, this should be called before all vram allocations * to protect retired page from abusing */
[Tao] The comment above should be also updated
[Guchun]yeah, will update it later.
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct amdgpu_device *adev) · line 2070
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3
-1-amdgpu_ras_recovery_init (adev); +r =amdgpu_ras_recovery_init (adev); +if (r) +goto init_failed;
[Tao] Are you sure "r != 0" equals to "bad gpu"? Other errors of recovery_init should not block gpu init. [Guchun]Good point. This should be addressed carefully.
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c
Quoted 140 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0if (adev->gmc.xgmi.num_physical_nodes >1 )
amdgpu_xgmi_add_device(adev); @@ -4133,8 +4135,20 @@ static int amdgpu_do_asic_reset(structamdgpu_hive_info*hive,
amdgpu_fbdev_set_suspend(tmp_adev, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14
-2-/* must succeed. */ -amdgpu_ras_resume (tmp_adev); +/* +* The CPU is BAD once faulty pages by ECC has +* reached the threshold, and ras recovery is +* scheduled. So add one check here to prevent +* recovery if it's one BAD GPU, and remind +* user to RMA this GPU. +*/ +if (!amdgpu_ras_check_bad_gpu (tmp_adev)) { +/* must succeed. */ +amdgpu_ras_resume (tmp_adev); +}else { +r = -EINVAL ; +goto out; +} /* Update PSP FW topology after reset */ if (hive && tmp_adev-Quoted 1 line
gmc.xgmi.num_physical_nodes > 1) @@ -4142,7 +4156,6 @@ static int
amdgpu_do_asic_reset(struct
amdgpu_hive_info*hive, } }drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-1- out:if (!r) {amdgpu_irq_gpu_reset_resume_helper (tmp_adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +1
-2@@ -62,8 +62,6 @@ const char *ras_block_string[] = { #define ras_err_str(i) (ras_error_string[ffs(i)]) #define ras_block_str(i) (ras_block_string[i]) -#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS1 -#define AMDGPU_RAS_FLAG_INIT_NEED_RESET2 #define RAS_DEFAULT_FLAGS (AMDGPU_RAS_FLAG_INIT_BY_VBIOS)/* inject address is 52 bits */ @@ -1817,6 +1815,7 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev)struct amdgpu_ras *con =amdgpu_ras_get_context (adev);struct ras_err_handler_data **data;uint32_t max_eeprom_records_len =0 ; +bool bad_gpu = false;int ret;if (con)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +7
-2@@ -1838,9 +1837,14 @@ int amdgpu_ras_recovery_init(struct amdgpu_device *adev) max_eeprom_records_len =amdgpu_ras_eeprom_get_record_max_length ();amdgpu_ras_validate_threshold (adev, max_eeprom_records_len); -ret =amdgpu_ras_eeprom_init (&con->eeprom_control); -if (ret) +ret =amdgpu_ras_eeprom_init (&con->eeprom_control, &bad_gpu); +/* +* we only fail this calling and halt booting when bad_gpu is true. +*/ +if (bad_gpu) { +ret = -EINVAL ; goto free; +}if (
con->eeprom_control.num_recs) {
ret =amdgpu_ras_load_bad_pages(adev); @@ -2189,3
+2193,16 @@ boolamdgpu_ras_need_emergency_restart(struct amdgpu_device *adev)return false;
}drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +12
-0+ +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev) { +struct amdgpu_ras *con =amdgpu_ras_get_context (adev); +bool bad_gpu = false; + +if (con && (con->bad_page_cnt_threshold !=0xFFFFFFFF )) +amdgpu_ras_eeprom_check_bad_gpu (&con->eeprom_control, +&bad_gpu); + +/* We are only interested in variable bad_gpu. */ +return bad_gpu; +}drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +6
-0@@ -31,6 +31,10 @@#include "ta_ras_if.h" #include "amdgpu_ras_eeprom.h" +#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS(0x1 << 0) +#define AMDGPU_RAS_FLAG_INIT_NEED_RESET(0x1 << 1) +#define AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV(0x1 << 2) +enum amdgpu_ras_block {AMDGPU_RAS_BLOCK__UMC =0 ,AMDGPU_RAS_BLOCK__SDMA ,@@ -493,6 +497,8 @@ void amdgpu_ras_suspend(struct amdgpu_device *adev); unsigned long amdgpu_ras_query_error_count(struct amdgpu_device *adev,bool is_ce); +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev); +/* error handling functions */ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev,struct
eeprom_table_record*bps, int pages); @@ -503,10
+509,14 @@ static inline intamdgpu_ras_reset_gpu(struct amdgpu_device *adev){
structamdgpu_ras*ras =amdgpu_ras_get_context(adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +7
-3-/* save bad page to eeprom before gpu reset, -* i2c may be unstable in gpu reset +/* +* Save bad page to eeprom before gpu reset, i2c may be unstable +* in gpu reset. +* +* Also, exclude the case when ras recovery issuer is +* eerprom page write. */ -if (in_task ()) +if (!(ras->flags & AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV) &&
in_task())
amdgpu_ras_reserve_bad_pages(adev);if (
atomic_cmpxchg(&ras->in_recovery, 0, 1)== 0) diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cindexa2c982b1eac6..96b63c026bad100644drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0
[Tao] It's better to split it into two patches, one is for ras and another one is for eeprom
[Guchun]I tried this, however, even if for ras change, eeprom change is needed as well. So I merged them both into one patch.
Quoted 72 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0@@ -46,6 +46,9 @@#define EEPROM_TABLE_HDR_VAL 0x414d4452 #define EEPROM_TABLE_VER
0x00010000drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-1+/* Bad GPU tag ‘BADG’ */ +#define EEPROM_TABLE_HDR_BAD 0x42414447 +/* Assume 2 Mbit size */ #define EEPROM_SIZE_BYTES 256000#define EEPROM_PAGE__SIZE_BYTES 256@@ -238,12 +241,14 @@ int amdgpu_ras_eeprom_reset_table(struct amdgpu_ras_eeprom_control *control) } -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control) +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu) {int ret =0 ;struct amdgpu_device *adev =to_amdgpu_device (control);unsigned char buff[EEPROM_ADDRESS_SIZE +EEPROM_TABLE_HEADER_SIZE ] = {0 };struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev);struct i2c_msg msg = {.addr= 0,
.flags=I2C_M_RD,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -251,6 +256,8 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) .buf= buff, }; +*bad_gpu = false; +/* Verify i2c adapter is initialized */ if (!adev->pm.smu_i2c.algo)return
-ENOENT;drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5
-0@@ -279,6 +286,11 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) DRM_DEBUG_DRIVER("Found existing EEPROM table with %d records", control->num_recs); +}else if ((ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && ( +hdr->header ==EEPROM_TABLE_HDR_BAD )) { +*bad_gpu = true; +DRM_ERROR ("Detect BAD GPU TAG in eeprom table and " +"GPU shall be RMAed.\n" ); }else {
DRM_INFO("Creating new EEPROM table");drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +13
-0@@ -375,6 +387,44 @@ static uint32_t __correct_eeprom_dest_address(uint32_t curr_address) return curr_address; } +int amdgpu_ras_eeprom_check_bad_gpu (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu) +{ +struct amdgpu_device *adev =to_amdgpu_device (control); +unsigned char buff[EEPROM_ADDRESS_SIZE + +EEPROM_TABLE_HEADER_SIZE ] = {0 }; +struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; +struct i2c_msg msg = { +.addr = control->i2c_address, +.flags =I2C_M_RD , +.len =EEPROM_ADDRESS_SIZE +EEPROM_TABLE_HEADER_SIZE , +.buf = buff, +};
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c static uint32_t · line 415
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +10
-0+int ret; + +*bad_gpu = false; + +/* read EEPROM table header */ +mutex_lock (&control->tbl_mutex); +ret =i2c_transfer (&adev->pm.smu_i2c, &msg,1 ); +if (ret <1 ) { +dev_err (adev->dev,"Failed to read EEPROM table header.\n" ); +goto err;
[Tao] One tab is missed
[Guchun]Correct this later.
Quoted 108 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-0+} + +__decode_table_header_from_buff (hdr, &buff[2 ]); + +if (hdr->header ==EEPROM_TABLE_HDR_BAD ) { +dev_warn(adev->dev, "Current GPU is BAD and should beRMAed.\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +9
-0+*bad_gpu = true; +} + +ret =0 ; +err: +mutex_unlock (&control->tbl_mutex); +return ret; +} + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control*control,
structeeprom_table_record
*records,
bool write,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -383,8 +433,10 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, int i, ret = 0;struct i2c_msg *msgs, *msg;unsigned char *buffs, *buff; +bool sched_ras_recovery = false;struct eeprom_table_record *record;struct amdgpu_device *adev =to_amdgpu_device (control); +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev); if (adev->asic_type != CHIP_VEGA20 && adev->asic_type !=
CHIP_ARCTURUS)
return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +19
-0@@ -402,6 +454,25 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, goto free_buff; } +/* +* If saved bad pages number exceeds the bad page threshod for +* the whole VRAM, update table header to mark one BAD GPU and +* schedule one ras recovery after eeprom write is done, this +* can avoid the missing for latest records. +* +* This new header will be picked up and checked in the bootup by +* ras recovery, which may break bootup process to notify user this +* GPU is bad and to RMA(Return Merchandise Authorization) this GPU. +*/ +if (write && (ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && +((control->num_recs + num) >= ras->bad_page_cnt_threshold)) { +dev_warn (adev->dev, +"Saved bad pages(%d) reaches threshold value(%d).\n" , +control->num_recs + num, ras- >bad_page_cnt_threshold); +control->tbl_hdr.header =EEPROM_TABLE_HDR_BAD ; +sched_ras_recovery = true; +} +/* In case of overflow just start from beginning to not lose newest records */
if (write && (control->next_addr+EEPROM_TABLE_RECORD_SIZE*
num >EEPROM_SIZE_BYTES))
control->next_addr=EEPROM_RECORD_START; @@ -482,6
+553,20 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control*control,
__update_tbl_checksum(control, records, num,old_hdr_byte_sum);
__update_table_header(control, buffs);drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +11
-0+ +if (sched_ras_recovery) { +/* +* Before scheduling ras recovery, assert the related +* flag first, which shall bypass common bad page +* reservation execution in amdgpu_ras_reset_gpu. +*/ +amdgpu_ras_get_context (adev)->flags |= +AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV ; + +dev_warn(adev->dev, "Conduct ras recovery due to bad"
+"page threshold reached.\n");drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0+amdgpu_ras_reset_gpu (adev); +} }else if (!__validate_tbl_checksum (control, records, num)) {DRM_WARN ("EEPROM Table checksum mismatch!" );/* TODO Uncomment when EEPROM read/write is relliable */ drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +5
-1@@ -77,9 +77,13 @@ struct eeprom_table_record {unsigned char mcumc_id; }__attribute__((__packed__)); -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu);int amdgpu_ras_eeprom_reset_table (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_check_bad_gpu (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu); + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control
Quoted 6 lines
*control,
structeeprom_table_record
*records,
bool write,
--
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset
Please see my comment about the patch organization
Quoted 218 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Thursday, July 23, 2020 11:39 AM
To: Zhou1, Tao <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset[AMD Public Use]
Thanks for your review, Tao.
Please check my comments after yours.Regards,
Guchun-----Original Message-----
From: Zhou1, Tao <[email protected]>
Sent: Thursday, July 23, 2020 10:51 AM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Clements, John <[email protected]>
Subject: RE: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/reset[AMD Official Use Only - Internal Distribution Only]
Quoted 5 lines
-----Original Message-----
From: Chen, Guchun <[email protected]>
Sent: Wednesday, July 22, 2020 11:14 AM
To: [email protected]; Deucher, Alexander <[email protected]>; Zhang, HawkingQuoted 2 lines
Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John
Quoted 14 lines
Cc: Chen, Guchun <[email protected]>
Subject: [PATCH 3/5] drm/amdgpu: conduct bad gpu check during bootup/resetDuring boot up, when init eeprom, RAS will check if the BAD GPU tag is saved or not. And will break boot up and notify user to RMA it.
At the meanwhile, when saved bad page count to eeprom exceeds threshold, one ras recovery will be triggered immediately, and bad gpu check will be executed and reset will fail as well to remind user.
User could set
bad_page_threshold= 0 as module parameter when probing driver to disable the bad feature check.Quoted 6 lines
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_device.c | 21 ++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 25 +++++- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h | 16 +++- .../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 87++++++++++++++++++-
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct · line 2068
Quoted 13 lines
.../gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h | 6 +- 5 files changed, 142 insertions(+), 13 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0@@ -2065,7 +2065,9 @@ static int amdgpu_device_ip_init(structamdgpu_device *adev) * Note: theoretically, this should be called before all vram allocations * to protect retired page from abusing */ [Tao] The comment above should be also updated [Guchun]yeah, will update it later.
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c static int amdgpu_device_ip_init(struct amdgpu_device *adev) · line 2070
Quoted 4 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +3
-1-amdgpu_ras_recovery_init (adev); +r =amdgpu_ras_recovery_init (adev); +if (r) +goto init_failed;[Tao] Are you sure "r != 0" equals to "bad gpu"? Other errors of
recovery_initshould not block gpu init.
[Guchun]Good point. This should be addressed carefully.Quoted 133 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-0if (adev->gmc.xgmi.num_physical_nodes >1 )
amdgpu_xgmi_add_device(adev); @@ -4133,8 +4135,20 @@ static int amdgpu_do_asic_reset(structamdgpu_hive_info*hive,
amdgpu_fbdev_set_suspend(tmp_adev, 0);drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +14
-2-/* must succeed. */ -amdgpu_ras_resume (tmp_adev); +/* +* The CPU is BAD once faulty pages by ECC has +* reached the threshold, and ras recovery is +* scheduled. So add one check here to prevent +* recovery if it's one BAD GPU, and remind +* user to RMA this GPU. +*/ +if (!amdgpu_ras_check_bad_gpu (tmp_adev)) { +/* must succeed. */ +amdgpu_ras_resume (tmp_adev); +}else { +r = -EINVAL ; +goto out; +} /* Update PSP FW topology after reset */ if (hive && tmp_adev-Quoted 1 line
gmc.xgmi.num_physical_nodes > 1) @@ -4142,7 +4156,6 @@ static int
amdgpu_do_asic_reset(struct
amdgpu_hive_info*hive, } }drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0
-1- out:if (!r) {amdgpu_irq_gpu_reset_resume_helper (tmp_adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +0
-1@@ -62,8 +62,6 @@ const char *ras_block_string[] = { #defineras_err_str (i) (ras_error_string[ffs(i)]) #define ras_block_str(i) (ras_block_string[i]) -#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS1 -#define
AMDGPU_RAS_FLAG_INIT_NEED_RESET2#defineRAS_DEFAULT_FLAGS(AMDGPU_RAS_FLAG_INIT_BY_VBIOS)/* inject address is 52 bits */
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +0
-0@@ -1817,6 +1815,7 @@ int amdgpu_ras_recovery_init(struct
amdgpu_device
*adev)
structamdgpu_ras*con =amdgpu_ras_get_context(adev); structras_err_handler_data**data;uint32_tmax_eeprom_records_len= 0;drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +1
-0+bool bad_gpu = false;int ret;if (con)@@ -1838,9 +1837,14 @@ int amdgpu_ras_recovery_init(struct
amdgpu_device
*adev)
max_eeprom_records_len=amdgpu_ras_eeprom_get_record_max_length();
amdgpu_ras_validate_threshold(adev, max_eeprom_records_len);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +8
-2-ret =amdgpu_ras_eeprom_init (&con->eeprom_control); -if (ret) +ret =amdgpu_ras_eeprom_init (&con->eeprom_control, &bad_gpu); +/* +* we only fail this calling and halt booting when bad_gpu is true. +*/ +if (bad_gpu) { +ret = -EINVAL ; goto free; +}if (con->eeprom_control.num_recs) { ret = amdgpu_ras_load_bad_pages(adev); @@ -2189,3 +2193,16 @@ bool amdgpu_ras_need_emergency_restart(struct
amdgpu_device*adev)return false;
}drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +9
-0+ +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev) {struct +amdgpu_ras *con =amdgpu_ras_get_context (adev);bool bad_gpu = false; + +if (con && (con->bad_page_cnt_threshold !=0xFFFFFFFF )) +amdgpu_ras_eeprom_check_bad_gpu (&con->eeprom_control, +&bad_gpu); + +/* We are only interested in variable bad_gpu. */ return bad_gpu; }drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +6
-0@@ -31,6 +31,10 @@#include "ta_ras_if.h" #include "amdgpu_ras_eeprom.h" +#define AMDGPU_RAS_FLAG_INIT_BY_VBIOS(0x1 << 0) #define +AMDGPU_RAS_FLAG_INIT_NEED_RESET(0x1 << 1) #define +AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV(0x1 << 2) +enum amdgpu_ras_block {AMDGPU_RAS_BLOCK__UMC =0 ,AMDGPU_RAS_BLOCK__SDMA ,@@ -493,6 +497,8 @@ void amdgpu_ras_suspend(struct amdgpu_device *adev); unsigned long amdgpu_ras_query_error_count(structamdgpu_device *adev,bool is_ce); +bool amdgpu_ras_check_bad_gpu (struct amdgpu_device *adev); +/* error handling functions */ int amdgpu_ras_add_bad_pages(struct amdgpu_device *adev, struct
eeprom_table_record*bps, int pages); @@ -503,10
+509,14 @@ static inline intamdgpu_ras_reset_gpu(struct amdgpu_device *adev){
structamdgpu_ras*ras =amdgpu_ras_get_context(adev);drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +7
-3-/* save bad page to eeprom before gpu reset, -* i2c may be unstable in gpu reset +/* +* Save bad page to eeprom before gpu reset, i2c may be unstable +* in gpu reset. +* +* Also, exclude the case when ras recovery issuer is +* eerprom page write. */ -if (in_task ()) +if (!(ras->flags & AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV) &&
in_task())
amdgpu_ras_reserve_bad_pages(adev);
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c
Quoted 6 lines
if (
atomic_cmpxchg(&ras->in_recovery, 0, 1)== 0) diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cb/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.cindexa2c982b1eac6..96b63c026bad100644drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0[Tao] It's better to split it into two patches, one is for ras and another one is for eeprom [Guchun]I tried this, however, even if for ras change, eeprom change is needed as well. So I merged them both into one patch.
[Tao] We may need to split it into more parts like this:
Part1: preparation for ras
Part2: preparation for eeprom
Part3: main part for ras
Part4: main part for eeprom
Quoted 182 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c line 46
Quoted 3 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0@@ -46,6 +46,9 @@#define EEPROM_TABLE_HDR_VAL 0x414d4452 #define
EEPROM_TABLE_VERQuoted 31 lines
0x00010000drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-1+/* Bad GPU tag ‘BADG’ */ +#define EEPROM_TABLE_HDR_BAD 0x42414447 +/* Assume 2 Mbit size */ #define EEPROM_SIZE_BYTES 256000#define EEPROM_PAGE__SIZE_BYTES 256@@ -238,12 +241,14 @@ int amdgpu_ras_eeprom_reset_table(struct amdgpu_ras_eeprom_control *control) } -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control) +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu) {int ret =0 ;struct amdgpu_device *adev =to_amdgpu_device (control);unsigned char buff[EEPROM_ADDRESS_SIZE +EEPROM_TABLE_HEADER_SIZE ] = {0 };struct amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev);struct i2c_msg msg = { .addr=0 ,.flags=
I2C_M_RD,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -251,6 +256,8 @@ int amdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control) .buf= buff, }; +*bad_gpu = false; +/* Verify i2c adapter is initialized */ if (!adev->pm.smu_i2c.algo)Quoted 2 lines
return
-ENOENT; @@ -279,6 +286,11 @@ intamdgpu_ras_eeprom_init(struct amdgpu_ras_eeprom_control *control)DRM_DEBUG_DRIVER("Foundexisting
Quoted 15 lines
EEPROM table with %d records",
control->num_recs);drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +5
-0+}else if ((ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && ( +hdr->header ==EEPROM_TABLE_HDR_BAD )) { +*bad_gpu = true; +DRM_ERROR ("Detect BAD GPU TAG in eeprom table and " +"GPU shall be RMAed.\n" ); }else {DRM_INFO ("Creating new EEPROM table" );@@ -375,6 +387,44 @@ static uint32_t
__correct_eeprom_dest_address(uint32_t curr_address)returncurr_address; }Quoted 1 line
+int amdgpu_ras_eeprom_check_bad_gpu(struct
amdgpu_ras_eeprom_controlQuoted 10 lines
*control,
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +7
-0+bool *bad_gpu) +{ +struct amdgpu_device *adev = to_amdgpu_device(control); unsigned char +buff[EEPROM_ADDRESS_SIZE + EEPROM_TABLE_HEADER_SIZE] = { 0 }; struct +amdgpu_ras_eeprom_table_header *hdr = &control->tbl_hdr; struct +i2c_msg msg = { .addr = control->i2c_address, .flags = I2C_M_RD, .len += EEPROM_ADDRESS_SIZE +
EEPROM_TABLE_HEADER_SIZE,
+.buf = buff,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c static uint32_t __correct_eeprom_dest_address(uint32_t curr_address)
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +10
-0+}; +int ret; + +*bad_gpu = false; + +/* read EEPROM table header */ +mutex_lock (&control->tbl_mutex); +ret =i2c_transfer (&adev->pm.smu_i2c, &msg,1 );if (ret <1 ) { +dev_err (adev->dev,"Failed to read EEPROM table header.\n" ); goto +err;[Tao] One tab is missed
[Guchun]Correct this later.Quoted 92 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +6
-0+} + +__decode_table_header_from_buff (hdr, &buff[2 ]); + +if (hdr->header == EEPROM_TABLE_HDR_BAD) { dev_warn(adev->dev, +"Current GPU is BAD and should beRMAed.\n");
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +9
-0+*bad_gpu = true; +} + +ret =0 ; +err: +mutex_unlock (&control->tbl_mutex); +return ret; +} + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control*control,
structeeprom_table_record
*records,
bool write,drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0@@ -383,8 +433,10 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, int i, ret = 0;struct i2c_msg *msgs, *msg;unsigned char *buffs, *buff; +bool sched_ras_recovery = false;struct eeprom_table_record *record;struct amdgpu_device *adev =to_amdgpu_device (control); +struct amdgpu_ras *ras =amdgpu_ras_get_context (adev); if (adev->asic_type != CHIP_VEGA20 && adev->asic_type !=
CHIP_ARCTURUS)
return 0;drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +19
-0@@ -402,6 +454,25 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control *control, goto free_buff; } +/* +* If saved bad pages number exceeds the bad page threshod for +* the whole VRAM, update table header to mark one BAD GPU and +* schedule one ras recovery after eeprom write is done, this +* can avoid the missing for latest records. +* +* This new header will be picked up and checked in the bootup by +* ras recovery, which may break bootup process to notify user this +* GPU is bad and to RMA(Return Merchandise Authorization) this GPU. +*/ +if (write && (ras->bad_page_cnt_threshold !=0xFFFFFFFF ) && +((control->num_recs + num) >= ras->bad_page_cnt_threshold)) { +dev_warn(adev->dev, "Saved bad pages(%d) reaches threshold +value(%d).\n", +control->num_recs + num, ras- >bad_page_cnt_threshold); +control->tbl_hdr.header =EEPROM_TABLE_HDR_BAD ; +sched_ras_recovery = true; +} +/* In case of overflow just start from beginning to not lose newest records */ if (write && (
control->next_addr+EEPROM_TABLE_RECORD_SIZE* num >EEPROM_SIZE_BYTES))control->next_addr=EEPROM_RECORD_START; @@ -482,6 +553,20 @@ int amdgpu_ras_eeprom_process_recods(structamdgpu_ras_eeprom_control*control,
__update_tbl_checksum(control, records, num, old_hdr_byte_sum);
__update_table_header(control, buffs);drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +11
-0+ +if (sched_ras_recovery) { +/* +* Before scheduling ras recovery, assert the related +* flag first, which shall bypass common bad page +* reservation execution in amdgpu_ras_reset_gpu. +*/ +amdgpu_ras_get_context (adev)->flags |= +AMDGPU_RAS_FLAG_SKIP_BAD_PAGE_RESV ; + +dev_warn(adev->dev, "Conduct ras recovery due to bad"
+"page threshold reached.\n");drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-0+amdgpu_ras_reset_gpu (adev); +} }else if (!__validate_tbl_checksum (control, records, num)) {
DRM_WARN("EEPROM Table checksum mismatch!");
/* TODO Uncomment when EEPROM read/write is relliable */ diff --gita/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.hb/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h
indexb272840cb069..4ccd139248a9100644drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h struct eeprom_table_record { unsigned char · line 85
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +3
-1@@ -77,9 +77,13 @@ struct eeprom_table_record { unsigned char mcumc_id; }__attribute__((__packed__)); -int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_init (struct amdgpu_ras_eeprom_control *control, +bool *bad_gpu);int amdgpu_ras_eeprom_reset_table (struct amdgpu_ras_eeprom_control *control); +int amdgpu_ras_eeprom_check_bad_gpu(struct
amdgpu_ras_eeprom_control
Quoted 10 lines
Quoted 10 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.h +2
-0*control, +bool *bad_gpu); + int amdgpu_ras_eeprom_process_recods(struct amdgpu_ras_eeprom_control*control,
structeeprom_table_record
*records,
bool write,
--
2.17.1
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 4/5 drm/amdgpu: restore ras flags when user resets eeprom
In reply to Guchun ChenRas flags needs to be cleaned as well when user requires one clean eeprom.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c | 2 ++ 1 file changed, 2 insertions(+)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +2 -0
@@ -371,6 +371,8 @@ static ssize_t amdgpu_ras_debugfs_eeprom_write(struct file *f, const char __userstruct amdgpu_device *adev = (struct amdgpu_device *)file_inode (f)->i_private;int ret; +amdgpu_ras_get_context (adev)->flags =RAS_DEFAULT_FLAGS ; + ret =amdgpu_ras_eeprom_reset_table (&adev->psp.ras.ras->eeprom_control);return ret ==1 ? size : -EIO ;
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
PATCH 5/5 drm/amdgpu: calculate actual size instead of hardcode size
In reply to Guchun ChenUse sizeof to get actual size.
--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2 -4
@@ -34,11 +34,9 @@/* * The 2 macros bellow represent the actual size in bytes that * those entities occupy in the EEPROM memory. -* EEPROM_TABLE_RECORD_SIZE is different than sizeof(eeprom_table_record) which -* uses uint64 to store 6b fields such as retired_page. */ -#define EEPROM_TABLE_HEADER_SIZE 20 -#define EEPROM_TABLE_RECORD_SIZE 24 +#define EEPROM_TABLE_HEADER_SIZE (sizeof(struct amdgpu_ras_eeprom_table_header)) +#define EEPROM_TABLE_RECORD_SIZE (sizeof(struct eeprom_table_record))#define EEPROM_ADDRESS_SIZE 0x2
Signature · 6 lines
2.17.1 _______________________________________________ amd-gfx mailing list [email protected] https://lists.freedesktop.org/mailman/listinfo/amd-gfx
Re: [PATCH 5/5] drm/amdgpu: calculate actual size instead of hardcode size
Quoted 12 lines · Guchun Chen
Use sizeof to get actual size.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-0
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c line 38
Quoted 5 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-2@@ -34,11 +34,9 @@/* * The 2 macros bellow represent the actual size in bytes that * those entities occupy in the EEPROM memory. -* EEPROM_TABLE_RECORD_SIZE is different than sizeof(eeprom_table_record) which -* uses uint64 to store 6b fields such as retired_page.
Did you find out the comment above was wrong ?
Andrey
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c line 42
Quoted 8 lines
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +2
-2*/ -#define EEPROM_TABLE_HEADER_SIZE 20 -#define EEPROM_TABLE_RECORD_SIZE 24 +#define EEPROM_TABLE_HEADER_SIZE (sizeof(struct amdgpu_ras_eeprom_table_header)) +#define EEPROM_TABLE_RECORD_SIZE (sizeof(struct eeprom_table_record))#define EEPROM_ADDRESS_SIZE 0x2
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx
RE: [PATCH 5/5] drm/amdgpu: calculate actual size instead of hardcode size
[AMD Public Use]
Hi Andrey,
Aha, thanks for your reminding, I ignore that comment. Let me update it later.
Regards,
Guchun
-----Original Message-----
From: Grodzovsky, Andrey <[email protected]>
Sent: Wednesday, July 22, 2020 10:26 PM
To: Chen, Guchun <[email protected]>; [email protected]; Deucher, Alexander <[email protected]>; Zhang, Hawking <[email protected]>; Li, Dennis <[email protected]>; Yang, Stanley <[email protected]>; Zhou1, Tao <[email protected]>; Clements, John <[email protected]>
Subject: Re: [PATCH 5/5] drm/amdgpu: calculate actual size instead of hardcode size
Quoted 17 lines · Guchun Chen
Use sizeof to get actual size.
Signed-off-by Guchun Chen <[email protected]>--- drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-)drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +0
-1@@ -34,11 +34,9 @@/* * The 2 macros bellow represent the actual size in bytes that * those entities occupy in the EEPROM memory. -* EEPROM_TABLE_RECORD_SIZE is different than
Quoted 2 lines
sizeof(eeprom_table_record)which
- * uses uint64 to store 6b fields such asretired_page.
Did you find out the comment above was wrong ?
Andrey
drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c line 41
Quoted 9 lines
*/drivers/gpu/drm/amd/amdgpu/amdgpu_ras_eeprom.c +3
-2-#define EEPROM_TABLE_HEADER_SIZE 20 -#define EEPROM_TABLE_RECORD_SIZE 24 +#define EEPROM_TABLE_HEADER_SIZE (sizeof(struct +amdgpu_ras_eeprom_table_header)) #define EEPROM_TABLE_RECORD_SIZE +(sizeof(struct eeprom_table_record))#define EEPROM_ADDRESS_SIZE 0x2
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx