From patchwork Mon Sep 14 21:09:56 2026 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit X-Patchwork-Submitter: James Hilliard X-Patchwork-Id: 3355 Return-Path: X-Original-To: noreply@patchwork.local Delivered-To: noreply@patchwork.local Received: from tor.lore.kernel.org (tor.lore.kernel.org [172.105.105.114]) by mxe881.netcup.net (Postfix) with ESMTPS id 482FE1C4D1B for ; Mon, 14 Sep 2026 23:15:25 +0200 (CEST) Authentication-Results: mxe881; dkim=pass header.d=gmail.com; spf=pass (sender IP is 172.105.105.114) smtp.mailfrom=linux-sunxi+bounces-25924-noreply=patchwork.local@lists.linux.dev smtp.helo=tor.lore.kernel.org Received-SPF: pass (mxe881: domain of lists.linux.dev designates 172.105.105.114 as permitted sender) client-ip=172.105.105.114; envelope-from=linux-sunxi+bounces-25924-noreply=patchwork.local@lists.linux.dev; helo=tor.lore.kernel.org; Received: from smtp.subspace.kernel.org (conduit.subspace.kernel.org [100.90.174.1]) by tor.lore.kernel.org (Postfix) with ESMTP id 1BE49295C1 for ; Mon, 14 Sep 2026 21:11:08 +0000 (UTC) Received: from localhost.localdomain (localhost.localdomain [127.0.0.1]) by smtp.subspace.kernel.org (Postfix) with ESMTP id 1F3BE4A341D; Mon, 14 Sep 2026 21:10:16 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="UCnIJroq" X-Original-To: linux-sunxi@lists.linux.dev Received: from mail-oi1-f176.google.com (mail-oi1-f176.google.com [209.85.167.176]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 7BB8D415F06 for ; Mon, 14 Sep 2026 21:10:11 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.167.176 ARC-Seal: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789420214; cv=none; b=CgMGByiqDWLgzvas/moS563fVZYjXpplSjkmKxnW6q1cUAPQdy2KstSzEUDq5T04ejCDktlzPgeQArDJm9G2oxTnczkxKh/msH81ooORufS1f3OTra/z8zrmi+PeEv+DopWUxw7Eb6epXmwRqAEnGkKO5IgsFopTHKFkZPn55Bw= ARC-Message-Signature: i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1789420214; c=relaxed/simple; bh=vtg+z9G0CuAdSRbu8cVWZ/uIsLBQGyskZGCo/vbBtVI=; h=From:Date:Subject:MIME-Version:Content-Type:Message-Id:References: In-Reply-To:To:Cc; b=u/nIWMb9Onk5B5FQzpOZNxdGege18QH6L6YqQN+Rg/mYyKRE1YJ22KW4pXKjVQb5byb7sE7VycwCahB05qAvde0HlXP7vFoq7c13Q0IzoX1Oft7XYEbdQxCuYHd/gmp/CXBt0X4kmoSnPUx9Red9AS8XXnEyjp2eot1BYd+xF7A= ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=UCnIJroq; arc=none smtp.client-ip=209.85.167.176 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Received: by mail-oi1-f176.google.com with SMTP id 5614622812f47-4c5373b1138so2039543b6e.3 for ; Mon, 14 Sep 2026 14:10:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20251104; t=1789420209; x=1790025009; darn=lists.linux.dev; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:from:to:cc:subject:date :message-id:reply-to:content-type; bh=y7F79jQI2v9SHnTcdbLyHrKpLzhqd8SwLgrM0tg4BfI=; b=UCnIJroqhI7H08tGrfDh66EOn2nGUdrKUa4TolpCooRRDaVOGdFBojrV3o1EtDgSg/ Havx+8RKfWQNBJl0er1XB8Mi3Ve8p6nRHcCBiP6ZdcHvFwp0CWp56if3zzjjWkzPWH62 gibU12pPryXDi/m3CU47KqmsTsjfWtsykwOd7MY604bnVEf8BruSSp8izm3HRqvn1VyB y8h2VD6vnd2e1ZBwsvqO102ckaC3eYvrQW8E3v0JV+5H3963Q/xN0q/k7zLMVvPBo2Rc Yj/EVRDfX6K0btxWDS0VoV/fidaUnNU+mSlhDXLTSxz2EipOgJpn9w+ISmjtOYG0vF2W +Mug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20260707; t=1789420209; x=1790025009; h=cc:to:in-reply-to:references:message-id:content-transfer-encoding :content-type:mime-version:subject:date:from:x-gm-gg :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to :content-type; bh=y7F79jQI2v9SHnTcdbLyHrKpLzhqd8SwLgrM0tg4BfI=; b=iKJzMhNQNN8oJhLXeQcwUw7MpMKgCz7NJpwFbhQVmM9Ef7muWbIGeQmJhAqVaPnsQX E62d6NAKfSHKoNgbOqbABtjVfyyiTZ13l7Y19e5lrksrTyVmT8UmwbuTGBMkjidrx0MY i58P5Pes6W0f9kY8mYrQ8GFWhv6C2EWY6W3CdvF6myyuNSbZbiNhYusY8SlGpCWc4Tb8 4XZD2gofd+UmOo5ya/NsO9kBRs6pvXlCACoHjg/kJVl3lgPnkS8Us7/qasBMcSJlnyyg EWnOmG4ytK2ryUiKHWoqLeXO/OMONk76HlS+tY0TSgb2K1xpVzXlK61+/a4fPpa9dA2V 92SA== X-Forwarded-Encrypted: i=1; AKwUvBxRFEQ2+ZzEWpSmHqG7qKDHD8610HjxpyHuSd37MdgZkdRVGvloks6nA1ITa7WBNVjQ2WpJUZGstXnKaA==@lists.linux.dev X-Gm-Message-State: AFuF++nmVKymUk9lc57GqkzNv3/fGqbTRR6qFUYJPGje9xpor9T6rC4y m+9HwB7C6vn+LxqpytBKaKU+fMT+wwBngF3NoGQmrGXtEZ4G1t6h4ZQ/ X-Gm-Gg: AYBFou3ESCW1ykue+9IHOdsex680GCVCBudHo/uf1jSNdxQVzqXtorYad7ILI9qBOyn BQGHY6bUs8t3queI/XHlA2lrMvPX11OprZCgxw2zWKAr3rwemFPaCPChxs3fk59mZ/MGU0ADyif SskCVKQL/DjKXtG18nrykE5hUSKWVU9asEQd22zzkcHFFZO/slMY5r3ejglapuCXieJGp9aPYKy lM07uxp64Vyi4Ab0tlpj1n3qyH+q5SmTws/iWIHOGmM3A64EBL/vedhCWwEVwqT4pjzyp1IHWjM FKGlBkrZQdTMf+tnnQsoPBhFNlmXlgcos7QxL6PB4el1f99uynQQ7kzMZYCuP+d/ZE15dm7wZO7 WStGJImGM0vmLRHcudH94gpu6u9cmOrOKoCXM0ss8mpSWkWncfSpHLNDoR4wSP/e15sEtLwoQX8 khPHLvF23xRVSka51K3OOC1RYr1mYAugHQ9AgisaByZzqhJY4oqKTQvLTvgHNzBE7/be3136YuX QiXMEv8LWuzutsgj43TwunamSouduCMvJGYiy9lhEoZ1L8emZ7tKwxgtl9obkyYk4SJL1EYGtK/ 6rL2L0R7Bz2ylpVdvZxf3uZNp2EiXYmZhe25kaTbQlvTgIpf+iR7GDQvRy3oiA5u5YXZ86n6Quc plSILofHSQRkkio0AfMHGNh5zWYCz3lDHpGQgT4A= X-Received: by 2002:a05:6820:2204:b0:6b1:5c8b:c0b9 with SMTP id 006d021491bc7-6c54008880fmr2485081eaf.3.1789420209405; Mon, 14 Sep 2026 14:10:09 -0700 (PDT) Received: from [127.0.1.1] (184-96-157-145.hlrn.qwest.net. [184.96.157.145]) by smtp.gmail.com with ESMTPSA id 006d021491bc7-6c45bfc3fe9sm4698544eaf.15.2026.09.14.14.10.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 14 Sep 2026 14:10:08 -0700 (PDT) From: James Hilliard Date: Mon, 14 Sep 2026 15:09:56 -0600 Subject: [PATCH v6 10/18] mtd: rawnand: sunxi: bound DMA batches by the user-data register bank Precedence: bulk X-Mailing-List: linux-sunxi@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Message-Id: <20260914-submit-sunxi-nand-vendor-oob-layout-v1-v6-10-2fb084fe0751@gmail.com> References: <20260914-submit-sunxi-nand-vendor-oob-layout-v1-v6-0-2fb084fe0751@gmail.com> In-Reply-To: <20260914-submit-sunxi-nand-vendor-oob-layout-v1-v6-0-2fb084fe0751@gmail.com> To: Miquel Raynal , Richard Weinberger , Vignesh Raghavendra , Rob Herring , Krzysztof Kozlowski , Conor Dooley , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , Maxime Ripard , Richard Genoud , Masahiro Yamada , Boris Brezillon , Brian Norris Cc: linux-mtd@lists.infradead.org, devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-sunxi@lists.linux.dev, linux-kernel@vger.kernel.org, James Hilliard X-Mailer: b4 0.15.2 X-MORS-Enabled: yes X-MORS-DOMAIN: patchwork.local X-MORS-HOSTING: hosting172546 X-MORS-USER: hosting172546 X-getmail-retrieved-from-mailbox: =?utf-8?q?INBOX?= The H6/H616 controller concatenates protected user data in a bank of 32 four-byte registers. The default layout can allocate more than 128 bytes across a page, even though each ECC step fits its individual length limit. DMA transfers then access beyond the user-data register bank. For example, a 16 KiB page with 1280 OOB bytes and BCH40/1024 has 160 user-data bytes, while a 16 KiB page with 1664 OOB bytes can allocate 512 bytes. Do not cap that allocation: changing the lengths would move ECC offsets and make existing pages written through PIO incompatible. Instead, split DMA transfers while retaining the original logical main-data and OOB offsets. Fill the first batch up to the register-bank limit. Transfer each remaining logical step separately using hardware slot and user-data register zero, repositioning the main column and spare-area base first. PAGE_OP generates the second and subsequent main-data columns from its hardware slot index, not relative to the initial column. Read and write probes on H616 confirm this behavior. A multi-slot batch starting partway through the page would therefore return to the wrong main-data column. Single-slot later batches avoid those internally generated column changes. Pages whose user data already fits retain the existing single-batch path. Pass the user-data register index directly to the protected-OOB helpers and program length fields from the batch's logical starting step. Collect each batch's ECC status and protected OOB before reusing the registers. Keep ECC accounting and trailing-OOB handling page-wide, remembering whether any batch found an erased chunk. Discard all partial DMA statistics before a PIO read retry. Use one page-read setup or program-begin command for the whole page. Only issue program-end after every batch and any trailing OOB transfer succeed. Do not retry a write in PIO after a batch has already been transferred. The allocator, free-OOB layout and raw-access callbacks are unchanged. Fixes: 54dcd6aa69db ("mtd: rawnand: sunxi: introduce maximize variable user data length") Signed-off-by: James Hilliard --- drivers/mtd/nand/raw/sunxi_nand.c | 284 ++++++++++++++++++++++---------------- 1 file changed, 163 insertions(+), 121 deletions(-) diff --git a/drivers/mtd/nand/raw/sunxi_nand.c b/drivers/mtd/nand/raw/sunxi_nand.c index e29638c828ca..18a4d1cc4246 100644 --- a/drivers/mtd/nand/raw/sunxi_nand.c +++ b/drivers/mtd/nand/raw/sunxi_nand.c @@ -54,6 +54,7 @@ #define NFC_REG_H6_RDATA_1 0x004C #define NFC_REG_A10_USER_DATA 0x0050 #define NFC_REG_H6_USER_DATA 0x0080 +#define NFC_H6_USER_DATA_REGS 32 #define NFC_REG_USER_DATA(nfc, x) (nfc->caps->reg_user_data + ((x) * 4)) #define NFC_REG_H6_USER_DATA_LEN 0x0070 /* A USER_DATA_LEN register can hold the length of 8 USER_DATA registers */ @@ -932,41 +933,47 @@ static u8 sunxi_nfc_user_data_sz(struct sunxi_nand_chip *sunxi_nand, int step) return sunxi_nand->user_data_bytes[step]; } +/* Keep the on-flash layout, but fit each DMA batch in the user-data bank. */ +static int sunxi_nfc_dma_batch_steps(struct nand_chip *nand, + int first_step, int end_step) +{ + struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand); + struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller); + unsigned int user_data_sz = 0; + int i; + + if (!nfc->caps->reg_user_data_len) + return end_step - first_step; + + /* + * PAGE_OP generates main-data columns from hardware slot zero, not + * from the initial column. Use only slot zero after the first batch, + * so the explicit column change selects each remaining logical step. + */ + if (first_step) + return 1; + + for (i = first_step; i < end_step; i++) { + user_data_sz += sunxi_nfc_user_data_sz(sunxi_nand, i); + if (user_data_sz > NFC_H6_USER_DATA_REGS * sizeof(u32)) + break; + } + + return i - first_step; +} + +/* PIO uses register zero; DMA concatenates user data within each batch. */ static void sunxi_nfc_hw_ecc_get_prot_oob_bytes(struct nand_chip *nand, u8 *oob, - int step, bool bbm, int page, + unsigned int reg_index, bool bbm, int page, unsigned int user_data_sz) { - struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand); struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller); u32 user_data; + unsigned int i; - if (!nfc->caps->reg_user_data_len) { - /* - * For A10, the user data for step n is in the nth - * REG_USER_DATA - */ - user_data = readl(nfc->regs + NFC_REG_USER_DATA(nfc, step)); - sunxi_nfc_user_data_to_buf(user_data, oob); - } else { - /* - * For H6 NAND controller, the user data for all steps is - * contained in 32 user data registers, but not at a specific - * offset for each step, they are just concatenated. - */ - unsigned int user_data_off = 0; - unsigned int reg_off; - u8 *ptr = oob; - unsigned int i; - - for (i = 0; i < step; i++) - user_data_off += sunxi_nfc_user_data_sz(sunxi_nand, i); - - user_data_off /= 4; - for (i = 0; i < user_data_sz / 4; i++, ptr += 4) { - reg_off = NFC_REG_USER_DATA(nfc, user_data_off + i); - user_data = readl(nfc->regs + reg_off); - sunxi_nfc_user_data_to_buf(user_data, ptr); - } + for (i = 0; i < user_data_sz / 4; i++) { + user_data = readl(nfc->regs + NFC_REG_USER_DATA(nfc, reg_index + i)); + sunxi_nfc_user_data_to_buf(user_data, oob + i * 4); } /* De-randomize the Bad Block Marker. */ @@ -1022,13 +1029,13 @@ static void sunxi_nfc_set_user_data_len(struct sunxi_nfc *nfc, } static void sunxi_nfc_hw_ecc_set_prot_oob_bytes(struct nand_chip *nand, - const u8 *oob, int step, + const u8 *oob, unsigned int reg_index, bool bbm, int page, unsigned int user_data_sz) { struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller); - struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand); u8 user_data[SUNXI_NFC_MAX_USER_DATA_SZ] = {}; + unsigned int i; /* Randomize the Bad Block Marker. */ if (bbm && (nand->options & NAND_NEED_SCRAMBLING)) { @@ -1037,33 +1044,10 @@ static void sunxi_nfc_hw_ecc_set_prot_oob_bytes(struct nand_chip *nand, oob = user_data; } - if (!nfc->caps->reg_user_data_len) { - /* - * For A10, the user data for step n is in the nth - * REG_USER_DATA - */ - writel(sunxi_nfc_buf_to_user_data(oob), - nfc->regs + NFC_REG_USER_DATA(nfc, step)); - } else { - /* - * For H6 NAND controller, the user data for all steps is - * contained in 32 user data registers, but not at a specific - * offset for each step, they are just concatenated. - */ - unsigned int user_data_off = 0; - const u8 *ptr = oob; - unsigned int i; - - for (i = 0; i < step; i++) - user_data_off += sunxi_nfc_user_data_sz(sunxi_nand, i); - - user_data_off /= 4; - for (i = 0; i < user_data_sz / 4; i++, ptr += 4) { - writel(sunxi_nfc_buf_to_user_data(ptr), - nfc->regs + NFC_REG_USER_DATA(nfc, user_data_off + i)); - } + for (i = 0; i < user_data_sz / 4; i++) { + writel(sunxi_nfc_buf_to_user_data(oob + i * 4), + nfc->regs + NFC_REG_USER_DATA(nfc, reg_index + i)); } - } static void sunxi_nfc_hw_ecc_update_stats(struct nand_chip *nand, @@ -1313,27 +1297,35 @@ static int sunxi_nfc_hw_ecc_read_extra_oob(struct nand_chip *nand, return 0; } -static int sunxi_nfc_hw_ecc_read_chunks_dma(struct nand_chip *nand, uint8_t *buf, - int oob_required, int page, - int nchunks) +static int sunxi_nfc_hw_ecc_read_batch_dma(struct nand_chip *nand, u8 *buf, + int oob_required, int page, + int first_step, int nchunks, + int *raw_mode) { bool randomized = nand->options & NAND_NEED_SCRAMBLING; struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand); struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller); struct mtd_info *mtd = nand_to_mtd(nand); struct nand_ecc_ctrl *ecc = &nand->ecc; - unsigned int corrected = mtd->ecc_stats.corrected; - unsigned int failed = mtd->ecc_stats.failed; + unsigned int reg_index = 0, user_data_sz; unsigned int max_bitflips = 0; - int ret, i, raw_mode = 0; + int ret, i; struct scatterlist sg; u32 status, pattern_found, wait; + if (first_step) { + ret = sunxi_nfc_read_column(nand, page, first_step * ecc->size, + NULL, 0); + if (ret) + return ret; + } + ret = sunxi_nfc_wait_cmd_fifo_empty(nfc); if (ret) return ret; - ret = sunxi_nfc_dma_op_prepare(nfc, buf, ecc->size, nchunks, + ret = sunxi_nfc_dma_op_prepare(nfc, buf + first_step * ecc->size, + ecc->size, nchunks, DMA_FROM_DEVICE, &sg); if (ret) return ret; @@ -1341,7 +1333,12 @@ static int sunxi_nfc_hw_ecc_read_chunks_dma(struct nand_chip *nand, uint8_t *buf sunxi_nfc_hw_ecc_enable(nand); sunxi_nfc_reset_user_data_len(nfc); for (i = 0; i < nchunks; i++) - sunxi_nfc_set_user_data_len(nfc, sunxi_nfc_user_data_sz(sunxi_nand, i), i); + sunxi_nfc_set_user_data_len(nfc, + sunxi_nfc_user_data_sz(sunxi_nand, first_step + i), i); + /* exec_op() restores the page's spare base during column changes. */ + if (first_step) + writel(mtd->writesize + sunxi_get_oob_offset(sunxi_nand, ecc, first_step), + nfc->regs + NFC_REG_SPARE_AREA(nfc)); sunxi_nfc_randomizer_config(nand, page, false); sunxi_nfc_randomizer_enable(nand); @@ -1374,15 +1371,17 @@ static int sunxi_nfc_hw_ecc_read_chunks_dma(struct nand_chip *nand, uint8_t *buf pattern_found = readl(nfc->regs + nfc->caps->reg_pat_found); pattern_found = field_get(NFC_ECC_PAT_FOUND_MSK(nfc), pattern_found); - for (i = 0; i < nchunks; i++) { - int data_off = i * ecc->size; - unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, i); - int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, i); + for (i = 0; i < nchunks; i++, reg_index += user_data_sz / 4) { + int logical_step = first_step + i; + int data_off = logical_step * ecc->size; + int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, logical_step); u8 *data = buf + data_off; u8 *oob = nand->oob_poi + oob_off; bool erased; int bitflips; + user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, logical_step); + bitflips = sunxi_nfc_hw_ecc_correct(nand, randomized ? data : NULL, oob_required ? oob : NULL, i, status, pattern_found, @@ -1397,40 +1396,64 @@ static int sunxi_nfc_hw_ecc_read_chunks_dma(struct nand_chip *nand, uint8_t *buf ret = sunxi_nfc_read_column(nand, page, mtd->writesize + oob_off, oob, ecc->bytes + user_data_sz); if (ret) - goto err_stats; + return ret; - sunxi_nfc_hw_ecc_get_prot_oob_bytes(nand, oob, i, !i, + sunxi_nfc_hw_ecc_get_prot_oob_bytes(nand, oob, reg_index, !logical_step, page, user_data_sz); } if (erased) - raw_mode = 1; + *raw_mode = 1; sunxi_nfc_hw_ecc_update_stats(nand, &max_bitflips, bitflips); } if (status & NFC_ECC_ERR_MSK(nfc)) { for (i = 0; i < nchunks; i++) { - int data_off = i * ecc->size; - unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, i); - int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, i); + int logical_step = first_step + i; + int data_off = logical_step * ecc->size; + int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, logical_step); u8 *data = buf + data_off; u8 *oob = nand->oob_poi + oob_off; if (!(status & NFC_ECC_ERR(i))) continue; + user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, logical_step); ret = sunxi_nfc_hw_ecc_read_error(nand, data, data_off, oob, mtd->writesize + oob_off, user_data_sz, &max_bitflips, page); if (ret < 0) - goto err_stats; + return ret; if (ret) - raw_mode = 1; + *raw_mode = 1; } } + return max_bitflips; +} + +static int sunxi_nfc_hw_ecc_read_chunks_dma(struct nand_chip *nand, u8 *buf, + int oob_required, int page, int nchunks) +{ + struct mtd_info *mtd = nand_to_mtd(nand); + unsigned int corrected = mtd->ecc_stats.corrected; + unsigned int failed = mtd->ecc_stats.failed; + unsigned int max_bitflips = 0; + int raw_mode = 0; + int first_step, batch_steps, ret; + + for (first_step = 0; first_step < nchunks; first_step += batch_steps) { + batch_steps = sunxi_nfc_dma_batch_steps(nand, first_step, nchunks); + ret = sunxi_nfc_hw_ecc_read_batch_dma(nand, buf, oob_required, page, + first_step, batch_steps, + &raw_mode); + if (ret < 0) + goto err_stats; + max_bitflips = max_t(unsigned int, max_bitflips, ret); + } + if (oob_required) { ret = sunxi_nfc_hw_ecc_read_extra_oob(nand, nand->oob_poi, NULL, !raw_mode, page); @@ -1761,69 +1784,88 @@ static int sunxi_nfc_hw_ecc_write_page_dma(struct nand_chip *nand, { struct sunxi_nfc *nfc = to_sunxi_nfc(nand->controller); struct sunxi_nand_chip *sunxi_nand = to_sunxi_nand(nand); + struct mtd_info *mtd = nand_to_mtd(nand); struct nand_ecc_ctrl *ecc = &nand->ecc; struct scatterlist sg; u32 wait; - int ret, i; + int first_step, batch_steps, ret, i; sunxi_nfc_select_chip(nand, nand->cur_cs); - ret = sunxi_nfc_wait_cmd_fifo_empty(nfc); - if (ret) - return ret; + for (first_step = 0; first_step < ecc->steps; first_step += batch_steps) { + unsigned int reg_index = 0; - ret = sunxi_nfc_dma_op_prepare(nfc, buf, ecc->size, ecc->steps, - DMA_TO_DEVICE, &sg); - if (ret) - goto pio_fallback; + batch_steps = sunxi_nfc_dma_batch_steps(nand, first_step, ecc->steps); + ret = sunxi_nfc_wait_cmd_fifo_empty(nfc); + if (ret) + return ret; - sunxi_nfc_reset_user_data_len(nfc); - for (i = 0; i < ecc->steps; i++) { - unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, i); - int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, i); - const u8 *oob = nand->oob_poi + oob_off; + ret = sunxi_nfc_dma_op_prepare(nfc, buf + first_step * ecc->size, + ecc->size, batch_steps, DMA_TO_DEVICE, &sg); + if (ret) { + /* Only retry before any part of the page has been transferred. */ + if (first_step) + return ret; + goto pio_fallback; + } - sunxi_nfc_hw_ecc_set_prot_oob_bytes(nand, oob, i, !i, page, - user_data_sz); - sunxi_nfc_set_user_data_len(nfc, user_data_sz, i); - } + sunxi_nfc_reset_user_data_len(nfc); + for (i = first_step; i < first_step + batch_steps; i++) { + unsigned int user_data_sz = sunxi_nfc_user_data_sz(sunxi_nand, i); + int oob_off = sunxi_get_oob_offset(sunxi_nand, ecc, i); + const u8 *oob = nand->oob_poi + oob_off; - ret = nand_prog_page_begin_op(nand, page, 0, NULL, 0); - if (ret) { - sunxi_nfc_dma_op_abort(nfc); - sunxi_nfc_dma_op_cleanup(nfc, DMA_TO_DEVICE, &sg); - return ret; - } + sunxi_nfc_hw_ecc_set_prot_oob_bytes(nand, oob, reg_index, !i, + page, user_data_sz); + sunxi_nfc_set_user_data_len(nfc, user_data_sz, i - first_step); + reg_index += user_data_sz / 4; + } - sunxi_nfc_hw_ecc_enable(nand); - sunxi_nfc_randomizer_config(nand, page, false); - sunxi_nfc_randomizer_enable(nand); + if (first_step) + ret = nand_change_write_column_op(nand, first_step * ecc->size, + NULL, 0, false); + else + ret = nand_prog_page_begin_op(nand, page, 0, NULL, 0); + if (ret) { + sunxi_nfc_dma_op_abort(nfc); + sunxi_nfc_dma_op_cleanup(nfc, DMA_TO_DEVICE, &sg); + return ret; + } - writel((NAND_CMD_RNDIN << 8) | NAND_CMD_PAGEPROG, - nfc->regs + NFC_REG_WCMD_SET); + /* exec_op() restores the page's spare base during column changes. */ + if (first_step) + writel(mtd->writesize + sunxi_get_oob_offset(sunxi_nand, ecc, first_step), + nfc->regs + NFC_REG_SPARE_AREA(nfc)); + sunxi_nfc_hw_ecc_enable(nand); + sunxi_nfc_randomizer_config(nand, page, false); + sunxi_nfc_randomizer_enable(nand); - wait = NFC_CMD_INT_FLAG; + writel((NAND_CMD_RNDIN << 8) | NAND_CMD_PAGEPROG, + nfc->regs + NFC_REG_WCMD_SET); - if (nfc->use_mdma) - wait |= NFC_DMA_INT_FLAG; - else - dma_async_issue_pending(nfc->dmac); + wait = NFC_CMD_INT_FLAG; - writel(NFC_PAGE_OP | NFC_DATA_SWAP_METHOD | - NFC_DATA_TRANS | NFC_ACCESS_DIR, - nfc->regs + NFC_REG_CMD); + if (nfc->use_mdma) + wait |= NFC_DMA_INT_FLAG; + else + dma_async_issue_pending(nfc->dmac); - ret = sunxi_nfc_wait_events(nfc, wait, false, 0); - if (ret) - sunxi_nfc_dma_op_abort(nfc); + writel(NFC_PAGE_OP | NFC_DATA_SWAP_METHOD | + NFC_DATA_TRANS | NFC_ACCESS_DIR, + nfc->regs + NFC_REG_CMD); - sunxi_nfc_randomizer_disable(nand); - sunxi_nfc_hw_ecc_disable(nand); + ret = sunxi_nfc_wait_events(nfc, wait, false, 0); + if (ret) + sunxi_nfc_dma_op_abort(nfc); - sunxi_nfc_dma_op_cleanup(nfc, DMA_TO_DEVICE, &sg); + sunxi_nfc_randomizer_disable(nand); + sunxi_nfc_hw_ecc_disable(nand); - if (ret) - return ret; + sunxi_nfc_dma_op_cleanup(nfc, DMA_TO_DEVICE, &sg); + + if (ret) + return ret; + } if (oob_required || (nand->options & NAND_NEED_SCRAMBLING)) { /* TODO: use DMA to transfer extra OOB bytes ? */