A federal appeals court revived a 2011 Federal Reserve rule on the fees banks can charge merchants when customers use their debit cards, a setback for retailers.