Skip to contentSkip to conversation

pci allocation resources problems on x86_64

Bug Mathieu Taillefumier 3 participants 8 messages Oct 24, 2008 – Oct 25, 2008

lore
Share & tools
lore
#0 lore raw

Hi,

The kernel does not seems to allocate the pci resources correctly on sony laptop (VGN-SZ71 series) when pcmcia slot is used.

I track down this bug for long now and I was able to identify where it is probably coming from. The laptop I am using possess a pcmcia card slot that I am using for a tvcard. The kernel is a x86_64 bits kernel (2.6.27.2) and the laptop has 4G of memory. In this case the initialization of the tv-card is pure garbage and the kernel oops after that. Now if I start the laptop with 2G and the same kernel then everything works fine. From that I can assume that the problem does not come from the tvcard driver and maybe not from the pcmcia driver (although I am not completely sure). After discussing on the dvb mailling list we arrive to the conclusion that the problem is probably coming from pci allocation ressources.

additional informations :

Configurations that work

- x86_64 kernels with 2G of memory
- x86 kernels with 4G of memory without PAE activated nor 64 bits resources allocations option activated.

Config that do not work

- x86_64 kernels with 4G of memory
- x86 kernels with 4G of memory with 64bits resources allocation activated (without PAE).

The problem can be reproduced with kernel-next and the kernel-git so far and probably with earlier kernels although I have try this

cat /proc/version
Linux version 2.6.27.2-intel-nogem (root@coesite) (gcc version 4.3.1 (GCC) ) #4 SMP Wed Oct 22 12:05:55 CEST 2008
the lspci output are in one of the files and different dmesg are included. the dmesg-diff give the differences between the working one and the buggy one.

I would like to help debugging this

Regards

Mathieu

#1 lore raw

Hi

Sorry, the first files iomem and ioports are not the right one (I used the mem option to check if it has any effects). Please find the right one in attachments. Sorry again for this mistake.

regards

Mathieu

Quoted 31 lines

Hi,

The kernel does not seems to allocate the pci resources correctly on sony laptop (VGN-SZ71 series) when pcmcia slot is used.

I track down this bug for long now and I was able to identify where it is probably coming from. The laptop I am using possess a pcmcia card slot that I am using for a tvcard. The kernel is a x86_64 bits kernel (2.6.27.2) and the laptop has 4G of memory. In this case the initialization of the tv-card is pure garbage and the kernel oops after that. Now if I start the laptop with 2G and the same kernel then everything works fine. From that I can assume that the problem does not come from the tvcard driver and maybe not from the pcmcia driver (although I am not completely sure). After discussing on the dvb mailling list we arrive to the conclusion that the problem is probably coming from pci allocation ressources.

additional informations :

Configurations that work

- x86_64 kernels with 2G of memory
- x86 kernels with 4G of memory without PAE activated nor 64 bits resources allocations option activated.

Config that do not work

- x86_64 kernels with 4G of memory
- x86 kernels with 4G of memory with 64bits resources allocation activated (without PAE).

The problem can be reproduced with kernel-next and the kernel-git so far and probably with earlier kernels although I have try this

cat /proc/version
Linux version 2.6.27.2-intel-nogem (root@coesite) (gcc version 4.3.1 (GCC) ) #4 SMP Wed Oct 22 12:05:55 CEST 2008
the lspci output are in one of the files and different dmesg are included. the dmesg-diff give the differences between the working one and the buggy one.

I would like to help debugging this

Regards

Mathieu

#2 lore raw
Quoted 52 lines · Mathieu Taillefumier

Hi

Sorry, the first files iomem and ioports are not the right one (I used the mem option to check if it has any effects). Please find the right one in attachments. Sorry again for this mistake.

regards

Mathieu

Quoted 42 lines

Hi,

The kernel does not seems to allocate the pci resources correctly on sony laptop (VGN-SZ71 series) when pcmcia slot is used.

I track down this bug for long now and I was able to identify where it is probably coming from. The laptop I am using possess a pcmcia card slot that I am using for a tvcard. The kernel is a x86_64 bits kernel (2.6.27.2) and the laptop has 4G of memory. In this case the initialization of the tv-card is pure garbage and the kernel oops after that. Now if I start the laptop with 2G and the same kernel then everything works fine. From that I can assume that the problem does not come from the tvcard driver and maybe not from the pcmcia driver (although I am not completely sure). After discussing on the dvb mailling list we arrive to the conclusion that the problem is probably coming from pci allocation ressources.

additional informations :

Configurations that work

- x86_64 kernels with 2G of memory
- x86 kernels with 4G of memory without PAE activated nor 64 bits resources allocations option activated.

Config that do not work

- x86_64 kernels with 4G of memory
- x86 kernels with 4G of memory with 64bits resources allocation activated (without PAE).

The problem can be reproduced with kernel-next and the kernel-git so far and probably with earlier kernels although I have try this

cat /proc/version
Linux version 2.6.27.2-intel-nogem (root@coesite) (gcc version 4.3.1 (GCC) ) #4 SMP Wed Oct 22 12:05:55 CEST 2008
the lspci output are in one of the files and different dmesg are included. the dmesg-diff give the differences between the working one and the buggy one.

I would like to help debugging this

Ultimately we need to do better at grabbing space for PCI allocations on x86. I was hoping we'd have some patches in 2.6.28 that would help here, but they weren't ready in time. Can you file a kernel.org bug for this problem with the files attached? I'll try to find time to put together some improvements...

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#3 lore raw

Hi,

Quoted 5 lines

Ultimately we need to do better at grabbing space for PCI allocations on x86. I was hoping we'd have some patches in 2.6.28 that would help here, but they weren't ready in time. Can you file a kernel.org bug for this problem with the files attached? I'll try to find time to put together some improvements...

yes sure. I will do that this weekend. I already grab some more information about this problem by reading docs. The problem seems related to iommu setup since it is not activated when the mem < 3G but activated when the mem>3G. I do not understand however how the kernel can report more that 4g of memory when there are only 4g installed (So I think that the bios is buggy too). I also remarked that the size of the /proc/mem file is not what it is intended to be. Another supprizing thing is these lines of the dmesg-not-working file

pci 0000:0a:00.0: BAR 0: got res [0x140000000-0x1400007ff] bus [0x140000000-0x14 00007ff] flags 0x20200
pci 0000:0a:00.0: BAR 0: moved to bus [0x140000000-0x1400007ff] flags 0x20200

it seems that the drivers setup some resources that way behind the available memory.

Regards

MAthieu

#4 lore raw
Quoted 16 lines · On Fri, Oct 24, 2008 at 1:10 PM, <[email protected]> wrote:

Hi,

Quoted 5 lines

Ultimately we need to do better at grabbing space for PCI allocations on x86. I was hoping we'd have some patches in 2.6.28 that would help here, but they weren't ready in time. Can you file a kernel.org bug for this problem with the files attached? I'll try to find time to put together some improvements...

yes sure. I will do that this weekend. I already grab some more information about this problem by reading docs. The problem seems related to iommu setup since it is not activated when the mem < 3G but activated when the mem>3G. I do not understand however how the kernel can report more that 4g of memory when there are only 4g installed (So I think that the bios is buggy too). I also remarked that the size of the /proc/mem file is not what it is intended to be. Another supprizing thing is these lines of the dmesg-not-working file

Quoted 6 lines

pci 0000:0a:00.0: BAR 0: got res [0x140000000-0x1400007ff] bus [0x140000000-0x14 00007ff] flags 0x20200
pci 0000:0a:00.0: BAR 0: moved to bus [0x140000000-0x1400007ff] flags 0x20200

it seems that the drivers setup some resources that way behind the available memory.

looks like that kernel do the right thing...

pci 0000:00:1f.3: BAR 0: got res [0xc2000000-0xc20000ff] bus [0xc2000000-0xc20000ff] flags 0x20200
pci 0000:00:1f.3: BAR 0: moved to bus [0xc2000000-0xc20000ff] flags 0x20200 pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02

pci 0000:00:1c.0:   IO window: 0x2000-0x2fff
pci 0000:00:1c.0:   MEM window: 0xf6000000-0xf7ffffff
pci 0000:00:1c.0:   PREFETCH window: 0x000000f0000000-0x000000f1ffffff
pci 0000:00:1c.1: PCI bridge, secondary bus 0000:06
pci 0000:00:1c.1:   IO window: 0x3000-0x3fff
pci 0000:00:1c.1:   MEM window: 0xf8000000-0xf9ffffff
pci 0000:00:1c.1:   PREFETCH window: 0x000000f2000000-0x000000f3ffffff
pci 0000:07:00.0: BAR 6: got res [0xf4000000-0xf401ffff] bus
[0xf4000000-0xf401ffff] flags 0x27200
pci 0000:00:1c.2: PCI bridge, secondary bus 0000:07
pci 0000:00:1c.2:   IO window: 0x4000-0x4fff
pci 0000:00:1c.2:   MEM window: 0xfa000000-0xfbffffff
pci 0000:00:1c.2:   PREFETCH window: 0x000000f4000000-0x000000f5ffffff
pci 0000:00:1c.3: PCI bridge, secondary bus 0000:08
pci 0000:00:1c.3:   IO window: 0x5000-0x5fff
pci 0000:00:1c.3:   MEM window: 0xc8000000-0xc9ffffff
pci 0000:00:1c.3:   PREFETCH window: 0x000000cc000000-0x000000cdffffff
pci 0000:09:04.0: CardBus bridge, secondary bus 0000:0a
pci 0000:09:04.0:   IO window: 0x006000-0x0060ff
pci 0000:09:04.0:   IO window: 0x006400-0x0064ff
pci 0000:09:04.0:   PREFETCH window: 0xc4000000-0xc7ffffff
pci 0000:09:04.0:   MEM window: 0x140000000-0x143ffffff
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:09
pci 0000:00:1e.0:   IO window: 0x6000-0x6fff
pci 0000:00:1e.0:   MEM window: 0xfc200000-0xfc2fffff
pci 0000:00:1e.0:   PREFETCH window: 0x000000c4000000-0x000000c7ffffff

...

bus: 00 index 0 io port: [0, ffff]
bus: 00 index 1 mmio: [0, ffffffffffffffff]
bus: 02 index 0 io port: [2000, 2fff]
bus: 02 index 1 mmio: [f6000000, f7ffffff]
bus: 02 index 2 mmio: [f0000000, f1ffffff]
bus: 02 index 3 mmio: [0, 0]
bus: 06 index 0 io port: [3000, 3fff]
bus: 06 index 1 mmio: [f8000000, f9ffffff]
bus: 06 index 2 mmio: [f2000000, f3ffffff]
bus: 06 index 3 mmio: [0, 0]
bus: 07 index 0 io port: [4000, 4fff]
bus: 07 index 1 mmio: [fa000000, fbffffff]
bus: 07 index 2 mmio: [f4000000, f5ffffff]
bus: 07 index 3 mmio: [0, 0]
bus: 08 index 0 io port: [5000, 5fff]
bus: 08 index 1 mmio: [c8000000, c9ffffff]
bus: 08 index 2 mmio: [cc000000, cdffffff]
bus: 08 index 3 mmio: [0, 0]
bus: 09 index 0 io port: [6000, 6fff]
bus: 09 index 1 mmio: [fc200000, fc2fffff]
bus: 09 index 2 mmio: [c4000000, c7ffffff]
bus: 09 index 3 io port: [0, ffff]
bus: 09 index 4 mmio: [0, ffffffffffffffff]
bus: 0a index 0 io port: [6000, 60ff]
bus: 0a index 1 io port: [6400, 64ff]
bus: 0a index 2 mmio: [c4000000, c7ffffff]
bus: 0a index 3 mmio: [140000000, 143ffffff]

because card bus need 0x4000000

resource before that already allocated to 0xce000000

and it can not use 0xd000000, because
pci 0000:00:02.0: found [8086/2a02] class 000300 header type 00
PCI: 0000:00:02.0 reg 10 64bit mmio: [fc000000, fc0fffff]
PCI: 0000:00:02.0 reg 18 32bit mmio: [d0000000, dfffffff]
PCI: 0000:00:02.0 reg 20 io port: [1800, 1807]

so it has to start from 0x140000000 after the RAM...

and that range it can not use that...

only solution could be

Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)

about pci_mem_start from 0xc1000000 instead of 0xc200000

__init void e820_setup_gap(void)

{
        unsigned long gapstart, gapsize, round;
        int found;

        gapstart = 0x10000000;
        gapsize = 0x400000;
        found  = e820_search_gap(&gapstart, &gapsize, 0, MAX_GAP_END);
#ifdef CONFIG_X86_64
        if (!found) {
                gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024;
                printk(KERN_ERR "PCI: Warning: Cannot find a gap in the 32bit "
                       "address range\n"
                       KERN_ERR "PCI: Unassigned devices with 32bit resource "
                       "registers may break!\n");
        }

#endif

        /*
         * See how much we want to round up: start off with
         * rounding to the next 1MB area.
         */
        round = 0x100000;
        while ((gapsize >> 4) > round)
                round += round;
        /* Fun with two's complement */
        pci_mem_start = (gapstart + round) & -round;

        printk(KERN_INFO
               "Allocating PCI resources starting at %lx (gap: %lx:%lx)\n",
               pci_mem_start, gapstart, gapsize);
}

please check if you can change memhole size in BIOS... if not, we can have more patch for it....make pci_mem_start more compact...

YH

#5 lore raw
Quoted 141 lines · Yinghai Lu
Quoted 16 lines · On Fri, Oct 24, 2008 at 1:10 PM, <[email protected]> wrote:

Hi,

Quoted 5 lines

Ultimately we need to do better at grabbing space for PCI allocations on x86. I was hoping we'd have some patches in 2.6.28 that would help here, but they weren't ready in time. Can you file a kernel.org bug for this problem with the files attached? I'll try to find time to put together some improvements...

yes sure. I will do that this weekend. I already grab some more information about this problem by reading docs. The problem seems related to iommu setup since it is not activated when the mem < 3G but activated when the mem>3G. I do not understand however how the kernel can report more that 4g of memory when there are only 4g installed (So I think that the bios is buggy too). I also remarked that the size of the /proc/mem file is not what it is intended to be. Another supprizing thing is these lines of the dmesg-not-working file

Quoted 6 lines

pci 0000:0a:00.0: BAR 0: got res [0x140000000-0x1400007ff] bus [0x140000000-0x14 00007ff] flags 0x20200
pci 0000:0a:00.0: BAR 0: moved to bus [0x140000000-0x1400007ff] flags 0x20200

it seems that the drivers setup some resources that way behind the available memory.

looks like that kernel do the right thing...

pci 0000:00:1f.3: BAR 0: got res [0xc2000000-0xc20000ff] bus [0xc2000000-0xc20000ff] flags 0x20200
pci 0000:00:1f.3: BAR 0: moved to bus [0xc2000000-0xc20000ff] flags 0x20200 pci 0000:00:1c.0: PCI bridge, secondary bus 0000:02

pci 0000:00:1c.0:   IO window: 0x2000-0x2fff
pci 0000:00:1c.0:   MEM window: 0xf6000000-0xf7ffffff
pci 0000:00:1c.0:   PREFETCH window: 0x000000f0000000-0x000000f1ffffff
pci 0000:00:1c.1: PCI bridge, secondary bus 0000:06
pci 0000:00:1c.1:   IO window: 0x3000-0x3fff
pci 0000:00:1c.1:   MEM window: 0xf8000000-0xf9ffffff
pci 0000:00:1c.1:   PREFETCH window: 0x000000f2000000-0x000000f3ffffff
pci 0000:07:00.0: BAR 6: got res [0xf4000000-0xf401ffff] bus
[0xf4000000-0xf401ffff] flags 0x27200
pci 0000:00:1c.2: PCI bridge, secondary bus 0000:07
pci 0000:00:1c.2:   IO window: 0x4000-0x4fff
pci 0000:00:1c.2:   MEM window: 0xfa000000-0xfbffffff
pci 0000:00:1c.2:   PREFETCH window: 0x000000f4000000-0x000000f5ffffff
pci 0000:00:1c.3: PCI bridge, secondary bus 0000:08
pci 0000:00:1c.3:   IO window: 0x5000-0x5fff
pci 0000:00:1c.3:   MEM window: 0xc8000000-0xc9ffffff
pci 0000:00:1c.3:   PREFETCH window: 0x000000cc000000-0x000000cdffffff
pci 0000:09:04.0: CardBus bridge, secondary bus 0000:0a
pci 0000:09:04.0:   IO window: 0x006000-0x0060ff
pci 0000:09:04.0:   IO window: 0x006400-0x0064ff
pci 0000:09:04.0:   PREFETCH window: 0xc4000000-0xc7ffffff
pci 0000:09:04.0:   MEM window: 0x140000000-0x143ffffff
pci 0000:00:1e.0: PCI bridge, secondary bus 0000:09
pci 0000:00:1e.0:   IO window: 0x6000-0x6fff
pci 0000:00:1e.0:   MEM window: 0xfc200000-0xfc2fffff
pci 0000:00:1e.0:   PREFETCH window: 0x000000c4000000-0x000000c7ffffff

...

bus: 00 index 0 io port: [0, ffff]
bus: 00 index 1 mmio: [0, ffffffffffffffff]
bus: 02 index 0 io port: [2000, 2fff]
bus: 02 index 1 mmio: [f6000000, f7ffffff]
bus: 02 index 2 mmio: [f0000000, f1ffffff]
bus: 02 index 3 mmio: [0, 0]
bus: 06 index 0 io port: [3000, 3fff]
bus: 06 index 1 mmio: [f8000000, f9ffffff]
bus: 06 index 2 mmio: [f2000000, f3ffffff]
bus: 06 index 3 mmio: [0, 0]
bus: 07 index 0 io port: [4000, 4fff]
bus: 07 index 1 mmio: [fa000000, fbffffff]
bus: 07 index 2 mmio: [f4000000, f5ffffff]
bus: 07 index 3 mmio: [0, 0]
bus: 08 index 0 io port: [5000, 5fff]
bus: 08 index 1 mmio: [c8000000, c9ffffff]
bus: 08 index 2 mmio: [cc000000, cdffffff]
bus: 08 index 3 mmio: [0, 0]
bus: 09 index 0 io port: [6000, 6fff]
bus: 09 index 1 mmio: [fc200000, fc2fffff]
bus: 09 index 2 mmio: [c4000000, c7ffffff]
bus: 09 index 3 io port: [0, ffff]
bus: 09 index 4 mmio: [0, ffffffffffffffff]
bus: 0a index 0 io port: [6000, 60ff]
bus: 0a index 1 io port: [6400, 64ff]
bus: 0a index 2 mmio: [c4000000, c7ffffff]
bus: 0a index 3 mmio: [140000000, 143ffffff]

because card bus need 0x4000000

resource before that already allocated to 0xce000000

and it can not use 0xd000000, because
pci 0000:00:02.0: found [8086/2a02] class 000300 header type 00
PCI: 0000:00:02.0 reg 10 64bit mmio: [fc000000, fc0fffff]
PCI: 0000:00:02.0 reg 18 32bit mmio: [d0000000, dfffffff]
PCI: 0000:00:02.0 reg 20 io port: [1800, 1807]

so it has to start from 0x140000000 after the RAM...

and that range it can not use that...

only solution could be

Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)

about pci_mem_start from 0xc1000000 instead of 0xc200000

__init void e820_setup_gap(void)

{
       unsigned long gapstart, gapsize, round;
       int found;

       gapstart = 0x10000000;
       gapsize = 0x400000;
       found  = e820_search_gap(&gapstart, &gapsize, 0, MAX_GAP_END);
#ifdef CONFIG_X86_64
       if (!found) {
               gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024;
               printk(KERN_ERR "PCI: Warning: Cannot find a gap in the 32bit "
                      "address range\n"
                      KERN_ERR "PCI: Unassigned devices with 32bit resource "
                      "registers may break!\n");
       }

#endif

       /*
        * See how much we want to round up: start off with
        * rounding to the next 1MB area.
        */
       round = 0x100000;
       while ((gapsize >> 4) > round)
               round += round;
       /* Fun with two's complement */
       pci_mem_start = (gapstart + round) & -round;

       printk(KERN_INFO
              "Allocating PCI resources starting at %lx (gap: %lx:%lx)\n",
              pci_mem_start, gapstart, gapsize);
}
Quoted 3 lines

please check if you can change memhole size in BIOS... if not, we can have more patch for it....make pci_mem_start more compact...

it is more worse

pci 0000:00:1c.3: scanning behind bridge, config 080800, pass 0
PCI: Scanning bus 0000:08
PCI: Fixups for bus 0000:08
PCI: bridge 0000:00:1c.3 io port: [5000, 5fff]
PCI: bridge 0000:00:1c.3 32bit mmio: [c8000000, c9ffffff]
PCI: bridge 0000:00:1c.3 64bit mmio pref: [cc000000, cdffffff]
PCI: Bus scan for 0000:08 returning with max=08

that c800000 and cc00000 is allocated by BIOS...

may need to set pci_mem_start to 0xc0000000.
and let 0x1f.3 to use 0xca00000

YH

#6 lore raw
In reply to Yinghai Lu

Hi,

Quoted 45 lines

and that range it can not use that...

only solution could be

Allocating PCI resources starting at c2000000 (gap: c0000000:20000000)

about pci_mem_start from 0xc1000000 instead of 0xc200000

__init void e820_setup_gap(void)

{
        unsigned long gapstart, gapsize, round;
        int found;

        gapstart = 0x10000000;
        gapsize = 0x400000;
        found  = e820_search_gap(&gapstart, &gapsize, 0, MAX_GAP_END);
#ifdef CONFIG_X86_64
        if (!found) {
                gapstart = (max_pfn << PAGE_SHIFT) + 1024*1024;
                printk(KERN_ERR "PCI: Warning: Cannot find a gap in the 32bit

"
"address range\n"
KERN_ERR "PCI: Unassigned devices with 32bit resource
"

                       "registers may break!\n");
        }

#endif

        /*
         * See how much we want to round up: start off with
         * rounding to the next 1MB area.
         */
        round = 0x100000;
        while ((gapsize >> 4) > round)
                round += round;
        /* Fun with two's complement */
        pci_mem_start = (gapstart + round) & -round;

        printk(KERN_INFO
               "Allocating PCI resources starting at %lx (gap: %lx:%lx)\n",
               pci_mem_start, gapstart, gapsize);
}
Quoted 2 lines

please check if you can change memhole size in BIOS... if not, we can have more patch for it....make pci_mem_start more compact...

It is not possible to modify anything from the bios at least nothing related to memory. So what do you suggest forcing the pci_mem_start. When I look at the dmesg file I find some open windows such as 0xc0000001 -> 0xdfffffff and 0xf0000001 -> to 0xfebffffff or something like that.

Mathieu

Quoted 8 lines

YH
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#7 lore raw
Quoted 6 lines · On Sat, Oct 25, 2008 at 5:43 AM, <[email protected]> wrote:

Hi,

It is not possible to modify anything from the bios at least nothing related to memory. So what do you suggest forcing the pci_mem_start. When I look at the dmesg file I find some open windows such as 0xc0000001 -> 0xdfffffff and 0xf0000001 -> to 0xfebffffff or something like that.

your BIOS only leave 0xc0000000 - 0xc7ffffff for kernel to allocate that to unassigned resource.

and your device for bus 0x0a need 0x4000000 for mmio and 0x4000000 for pref mmio.

but 1f.3 is not assigned resource too, need to make sure if it really need to assigned resource by OS too...
kernel should allocate 0xca00000 to it..., so simplely use quirks to disable it.

YH